Secondary content insertion in 360-degree video

ABSTRACT

A secondary content such as an advertisement may be inserted based on users&#39; interests in 360 degree video streaming. Users may have different interests and may watch different areas within a 360 degree video. The information about area(s) of 360 degree scenes that users watch the most may be used to select an ad(s) relevant to their interests. One or more secondary content viewports may be defined within a 360 degree video frame. Secondary content viewport parameter(s) may be tracked. For example, statistics of the user&#39;s head orientation for some time leading to tile presentation of the ad(s) may be collected. Secondary content may be determined based on the tracked secondary content viewport parameters).

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 62/376,227, filed on Aug. 17, 2016, the disclosure ofwhich is hereby incorporated by reference in its entirety.

BACKGROUND

Virtual 3600 video is a rapidly growing format emerging in the mediaindustry. 360° video is enabled by the growing availability of virtualreality (VR) devices. 360° video may provide the viewer a new sense ofpresence. When compared to rectilinear video (e.g., 2D or 3D), 360°video may pose difficult challenges on video processing and/or delivery.Enabling comfort and/or an immersive user experience may require highvideo quality and/or very low latency. The large video size of 360°video may be an impediment to delivering the 360° video to a largenumber of users.

HTTP streaming has become a dominant approach in commercial deployments.For instance, streaming platforms such as Apple's HTTP Live Streaming(HLS), Microsoft's Smooth Streaming (SS), and/or Adobe's HTTP DynamicStreaming (HDS) may use HTTP streaming as an underlying delivery method.A standard for HTTP streaming of multimedia content may enable astandard-based client to stream content from any standard-based server(e.g., thereby enabling interoperability between servers and clients ofdifferent vendors). MPEG Dynamic Adaptive Streaming over HTTP(MPEG-DASH) may be a universal delivery format that provides end userswith the best possible video experience by dynamically adapting tochanging network conditions. DASH may be built on top of the HTTP/TCP/IPstack. DASH may define a manifest format, Media Presentation Description(MPD), and segment formats for ISO Base Media File Format and MPEG-2Transport Streams.

SUMMARY

Methods, systems, and instrumentalities are disclosed for insertingsecondary content into a 360 degree video. Secondary content, such as anadvertisement may be inserted based on users' interests in 360 degreevideo streaming. Users may have different interests and may watchdifferent areas within a 360 degree video. The information about area(s)of 360 degree scenes that users watch the most may be used to select asecondary content relevant to their interests. One or more secondarycontent viewports may be defined within a 360 degree video frame.Statistics of the user's head orientation for some time leading to thepresentation of the secondary content may be collected. The secondarycontent may be determined based on statistics of the user's headorientation.

Statistics of user's head orientation may be collected over the entirestreaming session, over a particular period, or over specific timeintervals. Various media presentation description (MPD) elements such asdescriptor or event at MPD level, Period level or the Adaptation Setlevel may be used to activate the tracking of user's head orientationstatistics information. Orientation statistics may be used fordetermining secondary contents. For example, a video streaming clientmay use orientation statistics for app-based secondary contentinsertion. The video streaming client may send back orientationstatistics to a secondary content decision server for server-basedsecondary content insertion. The secondary content decision server mayupdate the MPD by inserting the corresponding secondary content based onorientation statistics. For example, the secondary content decisionserver may select different advertisements in response to orientationstatistics which indicate viewer interest in different secondary contentviewports.

Secondary contents may be inserted in a 360 degree video. For example,the 360 degree video may include one or more secondary contentviewports, and their respective secondary content viewport locations maybe received. For example, a MPD file may be received that may define oneor more secondary content viewports by specifying their correspondinghorizontal and/or vertical position of a reference point of thesecondary content viewport, a width and/or height of the secondarycontent viewport, and/or a secondary content viewport identifier. Arequest for tracking one or more secondary content viewport viewingparameters for the secondary content viewport(s) may be received. Forexample, the MPD file may indicate that the view counts, view durationand/or viewing orientation statistics associated with the secondarycontent viewports may be tracked. The request may indicate that thesecondary content viewport viewing parameter may be tracked periodically(e.g., by specifying a secondary content viewport classificationfrequency), over a period, and/or over a particular time period.

In response to the request, the secondary content viewport viewingparameter for the secondary content viewport may be tracked based on thesecondary content viewport location associated with the secondarycontent viewport and a viewer's viewing orientation. For example,whether the viewer's viewing orientation falls within the secondarycontent viewport may be determined, and the secondary content viewportviewing parameter may be updated accordingly. A secondary content may beselected, and a secondary content insertion location may be determinedbased on the tracked secondary content viewport viewing parameter. Forexample, the most viewed secondary content viewport may be determinedbased on the tracked secondary content viewport viewing parameter. Thesecondary content may be selected based on the content associated withthe most viewed secondary content viewport. A secondary contentinsertion position, a size and/or a secondary content insert type may bedetermined based on the secondary content viewport viewing parameter(s).

BRIEF DESCRIPTION OF THE DRAWINGS

A more detailed understanding of the embodiments disclosed herein may behad from the following description, given by way of example inconjunction with the accompanying drawings.

FIG. 1 depicts an example media presentation description (MPD)hierarchical data model.

FIG. 2 depicts an example portion of 360° video displayed on a headmounted device (HMD).

FIG. 3 shows a simplified example method for presenting 360 degree videobased on orientation.

FIG. 4 shows an example streaming video session with pre-, mid andpost-roll ads.

FIG. 5 shows an example server-based ad insertion using MPD update.

FIG. 6 shows an example DASH presentation with ad insertion using remoteperiods.

FIG. 7 shows an example call flow for DASH presentation with adinsertion in an app-based architecture.

FIG. 8 shows example secondary content viewport definitions within a 360degree video.

FIG. 9 shows example components for supporting secondary contentinsertion using secondary content viewport information.

FIG. 10 shows example DASH signaling for secondary content insertionusing secondary content viewports.

FIG. 11 depicts an example viewport defined over several regions of acubemap projection.

FIG. 12 depicts an exemplary 360 degree video frame partitioned in a 3×3grid for gathering orientation statistics.

FIG. 13 shows an example screenshot of a 360 degree video with a 2D adshown in a head-up display (HUD) layer.

FIG. 14 shows an example streaming client keeping orientationstatistics, enabling system to make decisions using secondary contentviewports.

FIG. 15 shows an example of keeping statistics of orientation for usewith secondary content viewports.

FIG. 16 shows an example of keeping statistics of orientation over areasof a 360 degree video.

FIG. 17 shows an example of mapping a point in a sphere into a 2Dprojection of a 360 degree video frame.

FIG. 18 shows an example of mapping a point in a sphere into 2Dprojection of 360 degree video frame over which grid is defined.

FIG. 19 shows an example of viewing statistics on secondary contentviewports that may be used to decide which ad is presented in astreaming session.

FIG. 20 shows an example of selecting secondary content based onsecondary content viewport information.

FIG. 21 shows an example call flow of a DASH presentation with secondarycontent insertion using remote periods and secondary content viewports.

FIG. 22 shows an example of selecting a secondary content usingsecondary content viewports.

FIG. 23 shows an example call flow of a DASH presentation with app-basedsecondary content insertion using secondary content viewports.

FIG. 24 is an example block diagram of e-commerce application withsecondary content viewports.

FIG. 25 shows an example e-commerce application using secondary contentviewports.

FIG. 26 shows an example 360 degree video scene with areas highlightedby an e-commerce application.

FIG. 27 shows an example of additional information shown by anc-commerce application.

FIG. 28 shows an example video streaming system where secondary contentviewports may be used for video analytics.

FIG. 29A is a system diagram illustrating an example communicationssystem in which one or more disclosed embodiments may be implemented.

FIG. 29B is a system diagram illustrating an example wirelesstransmit/receive unit (WTRU) that may be used within the communicationssystem illustrated in FIG. 29A according to an embodiment.

FIG. 29C is a system diagram illustrating an example radio accessnetwork (RAN) and an example core network (CN) that may be used withinthe communications system illustrated in FIG. 29A according to anembodiment.

FIG. 29D is a system diagram illustrating a further example RAN and afurther example CN that may be used within the communications systemillustrated in FIG. 29A according to an embodiment.

DETAILED DESCRIPTION

A detailed description of illustrative embodiments will now be describedwith reference to the various Figures. Although this descriptionprovides a detailed example of possible implementations, it should benoted that the details are intended to be exemplary and in no way limitthe scope of the application.

In dynamic HTTP streaming, various bitrate alternatives of themultimedia content may be available at the server. The multimediacontent may include several media components (e.g., audio, video, text),each of which may have different characteristics. In MPEG-DASH, thecharacteristics may be described by a Media Presentation Description(MPD).

The MPD may be an XML document that includes metadata for a DASH clientto construct appropriate HTTP-URLs to access video segments (e.g., asdescribed herein) in an adaptive manner during streaming sessions. FIG.1 depicts an example MPD hierarchical data model. The MPD may describe asequence of Periods, where a consistent set of encoded versions of themedia content components may remain the same during a Period. A Periodmay be associated with a starting time and duration. A Period mayinclude one or more adaptation sets (e.g., AdaptationSet).

An AdaptationSet may represent a set of encoded versions of one or moremedia content components sharing one or more identical properties, suchas the language, the media type, the picture aspect ratio, the role, theaccessibility, the viewpoint, and/or the rating property. For instance,a first AdaptationSet may include different bitrates of the videocomponent of the same multimedia content. A second AdaptationSet mayinclude different bitrates of the audio component (e.g., lower qualitystereo and/or higher quality surround sound) of the same multimediacontent. An AdaptationSet may include multiple Representations.

A Representation may describe a deliverable encoded version of one orseveral media components, varying from other representations by bitrate,resolution, number of channels, and/or other characteristics. Arepresentation may include one or more segments. One or more attributesof a Representation element (e.g., such as @id, @bandwidth,@qualityRanking, and @dependencyId) may be used to specify one or moreproperties of the associated Representation.

A Segment may be the largest unit of data that can be retrieved with asingle HTTP request. A segment may be associated with a URL (e.g., anaddressable location on a server). The segment may be downloaded usingHTTP GET or HTTP GET with byte ranges.

A DASH client may parse a MPD XML document. The DASH client may select acollection of AdaptationSets suitable for its environment, for example,based on information provided in the AdaptationSet elements. Within anAdaptationSet the client may select a Representation. The client mayselect the Representation based on the value of @bandwidth attribute,client decoding capabilities, and/or client rendering capabilities. Theclient may download an initialization segment of the selectedRepresentation. The client may access content, for example, byrequesting entire Segments or byte ranges of Segments. When thepresentation has started, the client may continue consuming the mediacontent. For example, the client may request Media Segments and/or partsof Media Segments during the presentation. The client may play contentaccording to a media presentation timeline. The client may switch from afirst Representation to a second Representation, based on updatedinformation from the client's environment. The client may play thecontent continuously across two or more Periods. When the client isconsuming media contained in the Segments towards the end of theannounced media in the Representation, the Media Presentation may beterminated, a Period may be started, and/or the MPD may be re-fetched.

The MPD descriptor element. Descriptor, may be provided to theapplication to instantiate one or more description elements with theappropriate scheme information. One or more Descriptors such as contentprotection, role, accessibility, rating, viewpoint, frame packing,and/or UTC timing descriptor may include a (@schemeIdUri attribute toidentify the relative scheme.

A supplemental property descriptor (SupplementalProperty) may includemetadata that may be used by the DASH client for optimizing processing.An essential property descriptor (EssentialProperty) may includemetadata for processing the containing element.

A Role MPD element may be associated with a @schemeIdUri attribute foridentifying the role scheme employed to identify the role of the mediacontent component. One or more Roles may define and/or describe one ormore characteristics and/or structural functions of media contentcomponents. An Adaptation Set and/or a media content component may havemultiple assigned roles (e.g., in different schemes or within the samescheme).

FIG. 2 depicts an example portion of 360° video displayed on a headmounted device (HMD). When looking around or zooming, the part of the360° video presented to the user may change based on the feedbackprovided by the HMD or other types of user interface (e.g.,smartphones). A viewpoint may include a spatial region of the entire360° video. A viewpoint may be fully or partially presented to the userand may have different quality than other parts of the 360° video.

Devices capable of streaming 360 degree video may be classified incertain major categories: devices that may use a HMD for an immersiveexperience (e.g., Oculus Rift. HTC Vive, Samsung Gear VR), or devicesthat display a viewport of the 360 degree video (e.g., Cardboard,YouTube and Facebook apps for Android and iOS smartphones). User's headorientation may be determined. For example, sensors in the HMD andadditional tracking device(s) may be used to determine user's headorientation during playback. For example, sensors in the smartphone maybe used to obtain such information. Orientation information may beobtained frequently. For example, orientation information may beobtained every time a frame has been decoded and it is ready to be shownon the display. This may allow the client to closely track orientationand to reduce lag between orientation changes and viewport displayed onthe screen.

FIG. 3 shows a simplified example method for presenting 360 degreevideo. As shown, after fetching and parsing the session's MPD, mediaplayback may occur one ‘Period’ at a time. For a ‘Period’, segments maybe fetched. After decoding the 360 degree video frames, the currentorientation may be determined. The orientation information may be usedto present the viewport that may correspond to the current orientation.

Secondary content, such as advertisement, educational information,gaming information, may be included in the 360 degree video. Advertisingmay be a source of revenue for streaming video. In online video,advertisements may be shown before (pre-roll), during (mid-roll) andafter (post-roll) some video content is presented to the user. Ads maybe inserted (or “spliced”) into the main content or presentation. Adbreaks (e.g., “avails”) may include points in time during a videostreaming session where one or more ads may be spliced. While contentinsertion is described herein in the context of ad insertion, it shallbe appreciated that the technology described herein may apply othercontent insertion such as information or video content related to aproduct showcased in the main video, educational or training videos(e.g., after completing a class, a set of new topics may be given touser), video games (e.g., game play may adapt to challenges that userenjoys the most), video library browsing (e.g., suggesting relatedtitles based on whether user likes to actively look around, or whetheruser enjoys a more quiet viewing experience) or the like.

A secondary content decision server, such as an ad decision server, maydecide whether to present secondary content. If secondary content is tobe presented, which secondary content may be presented during asecondary content presentation period, such as an ad break, may bedetermined. For example, an ad server may serve ad content to streamingclients. An ad decision server may or may not be on the same physicalserver as the ad server.

Ads may be seamlessly inserted (e.g., spliced) via digital “cuemessages”. Cue messages may be used to indicate time (e.g., presentationstart time) and parameters (e.g., length) of an upcoming ad break. Forexample, cue messages may be sent in advance of the start time of the ad(e.g., 6 and 2 seconds before) and of the resume time of the maincontent, allowing streaming clients to prepare for splicing mediastreams. Cue messages may be sent in-band (e.g., in ‘emsg’ boxes withinmedia segments) or out-of-band (e.g., in the MPD as user-definedevents).

When a video ad is presented, there may be certain rules for thestreaming client that may alter user's control of the playback. Forexample, users may be prevented from skipping or fast-forwarding throughthe ad, thus ensuring that the user sees the ad (all or part of the ad).

Advertisers may use the capabilities of modern video streamingtechniques for targeted advertising. For example, ads may be customizedbased on users' interests, demographic information or viewing habits.For example, IP address, geo-location, HTTP cookies, web beacons, uniquedevice token (a.k.a., device ad identifier), user login state to sitessuch as Google and Facebook, device graphs (non-personal informationused to track site interaction across multiple devices), etc. may beused for targeted advertising.

Ads may be inserted in DASH video streaming. For example, a DASH clientmay operate in an WTRU as described herein. The WTRU may be or mayinclude a DASH client device. A DASH client may playback content and adslisted in the MPD. For example, ad servers may decide whichadvertisement content should be presented to streaming clients.

Main content and ad playback may be predetermined in DASH. For example,a streaming session may include a mix of main content and ads. The MPDfetched by streaming clients may list a number of DASH Periods, where aPeriod may represent main content or an ad. Ads to be shown and theorder in which they are shown may be pre-determined by the contentprovider at the time of MPD generation. A main presentation may bepartitioned into multiple periods, and multiple ads may be shown in anad break. A simple example of a two-part main presentation, with pre-,mid- and post-roll ads, is shown in FIG. 4.

TABLE 1 Simplified sample MPD of a video session with pre-, mid- andpost-roll ads. <MPD>  <!-- Ad #1, pre-roll -->  <Period id=“Ad_1”>  <AdaptationSet mimeType=“video/mp4” codecs=“avc1.42c01e”segmentAlignment=“true”>    <Representation @id=“0” width=“1920”height=“1080” bandwidth=“127597”>     <BaseURL>ad_pre_roll.mp4</BaseURL>   </Representation>   </AdaptationSet>  </Period>  <!-- Mainpresentation, part 1 -->  <Period id=“Main_part_1”>   <AdaptationSetmimeType=“video/mp4” codecs=“ave1.42c01e” segmentAlignment=“true”>   <Representation @id=“0” width=“1920” height=“1080”bandwidth=“226597”>     <BaseURL>Main_content_part_1.mp4</BaseURL>   </Representation>   </AdaptationSet>  </Period>  <!-- Ad #2, mid-roll-->  <Period id=“Ad_2”>   <Adaptation Set mimeType=“video/mp4”codecs=“ave1.42c01e” segmentAlignment=“true”>    <Representation @id=“0”width=“1920” height=“1080” bandwidth=“127597”>    <BaseURL>ad_mid_roll.mp4</BaseURL>    </Representation>  </AdaptationSet>  </Period>  <!-- Main presentation, part 2 --> <Period id=“Main_part_2”>   <AdaptationSet mimeType=“video/mp4”codecs=“ave1.42c01e” segmentAlignment=“true”>    <Representation @id=“0”width=“1920” height=“1080” bandwidth=“226597”>    <BaseURL>Main_content_part_2.mp4</BaseURL>    </Representation>  </AdaptationSet>  </Period>  <!-- Ad #3, post-roll -->  <Periodid=“Ad_3”>   <AdaptationSet mime Type=“video/mp4” codecs=“ave1.42c01e”segmentAlignment=“true”>    <Representation @id=“0” width=“1920”height=“1080” bandwidth=“127597”>    <BaseURL>ad_post_roll.mp4</BaseURL>    </Representation>  </AdaptationSet>  </Period> </MPD>

A system may create a custom MPD for a streaming client (or for a groupof streaming clients). The MPD may be updated during a streamingsession. For example, synchronous updates may be performed via the‘minimumUpdatePeriod’ element on the MPD, which may indicate to theclient that it should re-fetch the MPD periodically (e.g., twice perminute). Asynchronous updates may be triggered by one or more events,prompting streaming clients to update their MPD(s). Such event mayinclude, but not limited to, MPD validity expiration event (e.g., whereMPD may be re-fetched upon reaching the expiration time), MPD patch(e.g., where a patch may be applied to the MPD already fetched by theclient), and MPD update event (where an entirely new MPD may bere-fetched). Events may be indicated in the MPD (e.g., using the‘InbandEventStream’ element) or in-band (e.g., as ISO-BMFF boxes of type‘emsg’).

The streaming clients may communicate with a secondary content serverfor secondary content insertion. For example, ad(s) may be resolvedduring playback, with an ad corresponding to a new ‘Period’ element inthe MPD. Streaming clients may use MPD update and/or remote new‘Periods’ for obtaining ‘Periods’ with ads.

FIG. 5 shows an example server-based ad insertion using MPD update. Anupdated MPD may be fetched by the client (e.g., synchronously and/orasynchronously) to determine new period(s) that correspond to ad(s).Multiple ads may result in multiple periods. After the ads have beenshown, a new period may be used to continue presenting the main content.A call flow of ad insertion using MPD update is shown in FIG. 5. The MPDmay be updated one or more times during playback. As described herein,the MPD may be updated periodically (synchronous), and/or when a DASHevent occurs or when the MPD expires (asynchronous). A sample MPD beforeupdate is shown in Table 2. As shown, the client may update its MPDevery 10 seconds. A sample MPD after update is shown in Table 3.

TABLE 2 Simplified example MPD before ad insertion update. <MPD . . .minimumUpdatePeriod=“PT10S”>  <!-- This is the first part of the maincontent. -->  <Period start=“PT0.00S” duration=“PT150.0S”id=“movie_period_1”>   <AdaptationSet mimeType=“video/mp4”codecs=“avc1.640828” . . . >    <BaseURL>video_1/</BaseURL>   <SegmentTemplate initialization=“$Bandwidth%/init.mp4v” . . . />   <Representation id=“v0” width=“320” height=“240” bandwidth=“250000”/>   <Representation id=“v1” width=“640” height=“480” bandwidth=“500000”/>  </AdaptationSet>  </Period>  <!-- An ad will be inserted here afterMPD is updated. -->  <!-- This is the second part of the main content.-->  <Period duration=“PT150.0S” id=“movie_period_2”>   <AdaptationSetmimeType=“video/mp4” codecs=“avc1.640828” . . . >   <BaseURL>video_2/</BaseURL>    <SegmentTemplateinitialization=“$Bandwidth%/init.mp4v” . . . />    <Representationid=“v0” width=“320” height=“240” bandwidth=“250000”/>    <Representationid=“v1” width=“640” height=“480” bandwidth=“500000”/>   </AdaptationSet> </Period> </MPD>

TABLE 3 Simplified example MPD updated for ad insertion. <MPD . . .minimumUpdatePeriod=“PT10S”>  <!-- This is the first part of the maincontent. -->  <Period start=“PT0.00S” duration=“PT150.0S”id=“movie_period_1”>   <AdaptationSet mimeType=“video/mp4”codecs=“avc1.640828” . . . >    <BaseURL>video_1</BaseURL>   <SegmentTemplate initialization=“$Bandwidth%/init.mp4v” . . . />   <Representation id=“v0” width=“320” height=“240” bandwidth=“250000”/>   <Representation id=“v1” width=“640” height=“480” bandwidth=“500000”/>  </AdaptationSet>  </Period>  <!-- After update, a new ad period isadded to the MPD -->  <!-- New period added after update: Mid-rolladvertisement -->  <Period duration=“PT30.00S” id=“ad_break_1”>  <AdaptationSet mimeType=“video/mp4” codecs=“ave1.42c01e” . . .>   <BaseURL>ad_campaign_1/</BaseURL>    <SegmentTemplateinitialization=“$Bandwidth%/init.mp4v” . . . />    <Representation@id=“ad0” width=“320” height=“240” bandwidth=“127597”>  </AdaptationSet>  </Period>  <!-- This is the second part of the maincontent -->  <Period duration=“PT150.0S” id=“movie_period_2”>  <AdaptationSet mimeType=“video/mp4” codecs=“avc1.640828” . . . >   <BaseURL>video_2/</BaseURL>    <SegmentTemplateinitialization=“$Bandwidth%/init.mp4v” . . . />    <Representationid=“v0” width=“320” height=“240” bandwidth=“250000”/>    <Representationid=“v1” width=“640” height=“480” bandwidth=“500000”/>   </AdaptationSet> </Period> </MPD>

A remote ‘Period’ may be used for updating MPD. Remote periods in an MPDmay include an xlink attribute with the information such as the URL fromwhich the period information may be fetched and/or the time at which theperiod information may be fetched. For example, the time at which theperiod information may be fetched may include “on load,” at the time theMPD is parsed and/or “on request,” when the client may need theinformation about the remote period.

FIG. 6 shows an example call flow of ad insertion using remote periods.The ad decision server may receive a request from the client, containinginformation about the main content that the client is playing back(e.g., using cue messages or using a DASH AssetIdentifier’descriptor),enabling the ad decision server to select an ad that corresponds to thecontent. The response to the ‘xlink’ request may be a ‘Period’ if one adis shown, or multiple ‘Periods’ if more than one ad is shown. The MPDmay contain default content in case the remote period cannot be resolved(e.g., ad decision server is busy or offline).

An ad decision server may customize the selected ad(s) based oninformation known about the requesting client. The client may include inthe HTTP request information about the user (e.g., gender and age), orthe ad server may get information about the client from the requestcontext (e.g., cookies or device ad identifiers in the HTTP request).For example, an ad agency may be running an ad campaign to promote a newproduct to young viewers watching some shows or videos online. Table 4shows an example MPD of a presentation with ad insertion using remoteperiods.

TABLE 4 Example MPD of a DASH presentation with ad insertion usingremote periods <MPD>  <!-- Movie -->  <Period start=“PT0.0S”duration=“PT600.6S” id=“movie_period_1”>   <AdaptationSetmimeType=“video/mp4” codecs--“avc1.640828” . . . >   <BaseURL>video_1/</BaseURL>    <SegmentTemplateinitialization=“$Bandwidth%/init.mp4v” . . . />    <Representationid=“v0” width=“320” height=“240” bandwidth=“250000”/>    <Representationid=“v1” width=“640” height=“480” bandwidth=“500000”/>   </AdaptationSet> </Period>  <!-- Mid-roll advertisement, passing base64url-coded SCTE 35to XLink resolver -->  <Period duration=“PT60.0S” id=“ad_break_1”  xlink:href=“https://adserv.com/avail.mpd?scte35-time=PT600.6S&scte35-cue=DAIAAAAAAAAAAAQAAZ_I0VniQAQAgBDVUVJQAAAAH+cAAAAAA %3D%3D”  xlink:actuate=“onRequest” >   <!-- Default content, replaced byelements from remote entity -->   <AdaptationSet mimeType=“video/mp4”codecs=“avc1.640828” . . . >    <BaseURLavailabilityTimeOffset=“INF”>default_ad/</BaseURL>    <SegmentTemplateinitialization=“$Band-width%/init.mp4v” . . . />    <Representationid=“v0” width=“320” height=“240” bandwidth=: 125000/>   </AdaptationSet> </Period>  <!-- Movie, cont'd -->  <Period duration=“PT600.0S”id=“movie_period_2”>   <AdaptationSet mimeType=“video/mp4”codecs=“ave1.640828” . . . >    <BaseURL>video_2/</Base URL>   <SegmentTemplate initialization=“$Bandwidth%/init.mp4v” . . . />   <Representation id=“v0” width=“320” height=“240” bandwidth=“250000”/>   <Representation id=“v1” width=“640” height=“480” bandwidth=“500000”/>  </AdaptationSet>  </Period> </MPD>

An ad may be represented by a separate MPD. The client may contact thead decision server to determine the content to be played when a cuemessage is received. As described herein, cue messages may be listed inthe MPD as user-defined events (in an EventStream element within aPeriod element) and/or may be sent in-band in ‘emsg’ boxes within mediasegments.

FIG. 7 shows an example call flow for DASH presentation with adinsertion in an app-based architecture. Upon receiving a cue message,the client may pass the cue information to an ad management module. Thead management module may communicate with the ad decision server toobtain a new MPD URL for the ad. Cue information may identify thecontent being played back by the client. The request sent to the addecision server may include information about the client (e.g., viewinghabits) and its context (e.g., HTTP cookies or device ad identifiers),enabling the ad decision server to select an appropriate ad. The DASHclient may get the ad MPD containing an ad period, pause presentation ofmain content, and start presenting the ad. Upon completion, the clientmay resume presentation of main content, e.g., as shown in the call flowin FIG. 7. If multiple ads are shown, multiple URLs may be obtained fromthe ad decision server, and/or the ad MPD may contain multiple periods.

A 360 degree video type and projection method may be signaled. Commonattributes may be specified for AdaptationSet, Representation, orSub-Representation. The @videoType attribute may indicate the video typesuch as spherical video, light field video or rectilinear video. Forspherical video, different projection formats and/or projection+layoutcombinations, such as equirectangular, cube-map (in combination withdifferent layouts) and pyramid map can be signaled as Common attribute@projection and @layout for AdaptationSet, Representation, and/orSub-Representation elements. Tables 5-8 shows example syntax of the‘@videoType’, ‘@projection’ and ‘@layout’ attributes.

TABLE 5 example common attributes and elements Element or Attribute NameUse Description Common attributes and elements @videoType O May specifythe video type in the form of an integer as specified in Table 6. If notpresent, the value is inferred to be “rectilinear”. @projection O Mayspecify the projection method used to project omnidirectional video intorectilinear video frame as specified in Table 6. If @videoType is equalto 2, the value of @projection is specified in Table 7. @layout O Mayspecify the layout of projected omnidirectional video in the form of aninteger. The value of @layout is specified in Table 8.

TABLE 6 example value of @videoType Value Geometry Type 0 Rectilinearvideo 1 Panorama video 2 Spherical video 3 Lightfield video . . . . . .

TABLE 7 example value of @projection Value Geometry Type 0Equirectangular 1 Cube 2 Offset cube 3 Squished sphere 4 Pyramid 5Cylinder . . . . . .

TABLE 8 example value of @layout @projection @layout Layout format 0 0Plate carree 1 Poles on the side (half height) 2 Poles on the side (fullheight) 1 3 Single row 4 2 × 3 5 180° . . . . . . . . .

The projection method used by the main content and ads may be different.For example, the main content may use cubemap. Some ads may useequirectangular and other ads may use pyramid projection. Table 9 showsa sample MPD of a 360 degree video presentation with a pre-roll and apost-roll ad. An ad may use a video type and/or projection methoddifferent from the main presentation.

TABLE 9 Sample simplified MPD of a 360 degree video presentation withadvertisements. <?xml version=“1.0” encoding=“UTF-8”?> <MPD> <ProgramInformation>   <Title>Example of a 360 degree video DASH MediaPresentation Description    with pre- and post-rolladvertisements<Title>  </ProgramInformation>  <!-- 360 degree Ad 1,pre-roll, Video type is equirectangular. -->  <Period id=“Ad_1”>  <AdaptationSet mimeType=“video/mp4” . . . videoType=2>   <Representation @id=“0” width=“4096” height=“2048”bandwidth=“327597”>     <BaseURL>360_degree_ad_pre_roll.mp4</BaseURL>   </Representation>   </AdaptationSet>  </Period>  </-- 360 degree Mainpresentation. Video type is cubemap 2x3. -->  <Period id=“Main”>  <AdaptationSet mimeType=“video/mp4” . . . videoType=2 projection=1layout=1>    <Representation @id=“0” width=“4096” height=“2048”bandwidth=“626597”>     <BaseURL>360_degree_main_content.mp4</BaseURL>   </Representation>   </AdaptationSet>  </Period>  </-- 360 degree Ad2, post-roll. Video type is pyramid. -->  <Period id=“Ad_2”>  <AdaptationSet mimeType=“video/mp4” . . . videoType=2 projection=4>   <Representation @id=“0” width=“4096” height=“2048”bandwidth=“327597”>     <BaseURL>360_degree_ad_post_roll.mp4</BaseURL>   </Representation>   </AdaptationSet>  </Period> </MPD>

360 degree video may or may not be 360 degrees. For example, as usedherein a 360 degree video may include spherical video, virtual reality(VR) video, panorama video, omnidirectional video, and/or immersivevideo such as light field video (e.g., 6 degree of freedom) and pointcloud video. Secondary content may or may not be 360 degree video. Forexample, secondary content may include 2D video and/or a mix of 2D orrectilinear video and 360 degree video content.

A secondary content may be inserted based on users' interests in 360degree video streaming. 360 degree video may allow content producers topresent scenes where multiple areas may call the attention of viewers.Users may have different interests and may watch different areas withina 360 degree video. Secondary content viewports may be used to indicatecertain areas of interests within a 360 degree video. For example, an adviewport may be a secondary content viewport that may be used toindicate certain areas of interests within a 360 degree video tofacilitate ad insertions. The information about area(s) of 360 degreescenes that users watch the most may be used to select secondarycontent(s), such as an ad(s), product information, backgroundinformation, relevant to their interests.

FIG. 8 shows example secondary content viewport definitions within a 360degree video frame. It shows a street scene in a 360 degree video withthree secondary content viewports: secondary content viewport #1 definesan area that contains an architecturally interesting building, secondarycontent viewports #2 and #3 define areas that contain stores. Whilewatching this video, some users may be interested in buildingarchitecture (SC viewport #1), whereas others may be interested instores and shopping (SC viewports #2 and #3). The streaming client maykeep statistics of the user's head orientation for some time leading tothe presentation of the secondary content. Secondary content, such as adcontent, may be determined based on user's viewing statistics of thesecondary content viewport(s). For example, with this information(and/or other information such as viewing habits, user profile, etc.),an ad decision server may decide to show the first group of users an adabout travel, while it may show the second group an ad about shopping. Astreaming client (e.g., video streaming client), as used herein, mayinclude a DASH client application, a DASH compliant video streamingapplication, a video streaming device such as the WTRU as describedherein, and/or any other application or device that may render videodata.

Secondary content viewport parameter(s) may be tracked for the secondarycontent viewport(s) in the 360 degree video. Secondary content viewportparameters may include statistics of user's head orientation. Statisticsof user's head orientation may be collected over the entire streamingsession, over a particular period, or over specific time intervals.Secondary content viewport information may be determined based on theuser's head orientation statistics. Secondary content viewportinformation may include the locations of the secondary content viewportsand their respective secondary content viewport identifiers.

A secondary content viewport location may be indicated via a horizontaland/or vertical position of a reference point (e.g., top left corner) ofthe secondary content viewport, and/or a width and/or height of thesecondary content view port. Various MPD elements such as descriptor orevent at MPD level, Period level or the Adaptation Set level may be usedto activate the tracking of user's head orientation statisticsinformation.

The client may track secondary content viewport parameter(s), forexample, based on user orientation statistics and/or secondary contentviewport information, to make secondary content insertion decisions. Theclient may pass secondary content viewport statistics to another moduleon the device for app-based ad insertion. The client may send backsecondary content viewport statistics and/or secondary content viewportinformation to the secondary content decision server for server-basedsecondary content insertion. The secondary content decision server mayupdate the MPD by inserting the corresponding secondary content based onorientation statistics and/or secondary content viewport information.For example, the secondary content decision server may select differentadvertisements for users that viewed different secondary contentviewports as shown in FIG. 8. For example, the secondary contentdecision server may select different advertisements for display indifferent secondary content viewports shown in FIG. 8.

A location for inserting secondary content may be determined based onsecondary content viewport information. Most viewed secondary contentviewport(s) may be determined based on user orientation statistics. Forexample, the streaming client may send orientation statistics to an adtracking server to identify the most viewed areas of a 360 degree video.This data may be aggregated and, may be used to auction the most viewedad viewports(s) at higher price, or to link high profit ads to multiplesecondary content viewports to increase hit rate.

FIG. 9 shows an example process for enabling ad insertion based onusers' interests and secondary content viewports in 360 degree videostreaming. As shown, information associated with secondary contentviewport may be signaled. Statistics related to secondary contentviewport(s) may be kept. Statistics of user's head orientation over a‘Period’, part of a ‘Period’ and/or over the entire streaming sessionmay be recorded. Statistics over part of a period may be used for adinsertion, for example if viewing of a secondary content viewport lastsfor a limited amount of time. Statistics over an entire streamingsession may be used to identify areas of a 360 video that are mostinteresting to users. Secondary content decisions may be made based onthe statistics. Statistics related to secondary content viewport(s) maybe reported.

While the description of secondary content viewports herein is providedin the context of a DASH streaming system, the concepts described mayalso be achieved by other means in a streaming system. For example,secondary content viewport signaling may be provided to the client usingone or more of metadata obtained from other modules in the system (e.g.,from a remote server), metadata that may be embedded in the media filesreceived from the content server (e.g., within the free space (‘free’ or‘skip’) box of the ISO-BMFF), or out-of-band means (e.g., using acontrol channel) where secondary content viewport information may beobtained by the client from the content server or other modules in thestreaming system.

Secondary content viewport information may be sent using DASH signalingin a DASH streaming system. FIG. 10 shows example DASH signaling forsecondary content insertion using secondary content viewports. As shown,a streaming client may receive a request to track secondary contentviewport viewing parameter(s) such as viewing statistics. The secondarycontent viewport position(s) and time for tracking viewing parameter(s)related to the secondary content viewport(s) may be determined based onthe request. Relations between secondary content viewports and secondarycontents for potential insertion may be determined. Secondary contentviewport viewing parameter(s) may be reported.

For example, an indication to start keeping statistics during an entireperiod may be sent to a client. The indication may include one or moreof, but not limited to, an ‘InbandEventStream’ element, an ‘EventStream’element, a ‘SupplementalProperty’ element and/or the like. The‘InbandEventStream’ and ‘SupplementalProperty’ elements may beassociated with ‘AdaptationSet’. The element ‘EventStream’ may beassociated with ‘Period’. In response to receiving the indication tostart keeping statistics during an entire period, the client may startto keeping statistics during the entire period of the video.

An ‘InbandEventStream’ element may be used to indicate to the clientthat it is to start tracking secondary content viewport parameter(s).The ‘orientation stats keep’ scheme shown in Table 10 may be used tospecify this signal. Keeping statistics may start at the beginning andmay stop at the end of the containing Period element. A simplifiedexample MPD using ‘InbandEventStream’ to indicate keeping statisticsover a ‘Period’ is shown in Table 11.

TABLE 10 Example ‘Orientation stats keep’ scheme for'InbandEventStream’. Element or Attribute Name DescriptionInbandEventStream @schemeIdUri urn:mpeg:dash:orientationstatskeep:2016

TABLE 11 Simplified example MPD signaling ‘orientation stats keep’ overa ‘Period’ using ‘InbandEventStream’. <MPD>  <Period>   <AdaptationSet>   <!-- Keep statistics of user's head orientation over duration of thisperiod -->    <InbandEventStreamschemeIdUri=“urn:mpeg:dash:orientationstatskeep:2016” />    . . .  </AdaptationSet>   . . .  </Period> </MPD>

An ‘EventStream’ element may be used to indicate to the client that itis to start keeping statistics. The ‘EventStream’ elements may be usedat the ‘Period’ level. The ‘InbandEventStream’ elements may beassociated with ‘AdaptationSet’ or ‘Represention’ elements. The example‘orientation stats keep’ scheme shown in Table 12 may be used to specifythis signal. Due to the scope of ‘EventStream’, statistics collectionmay start at the beginning and may stop at the end of the containing‘Period’ element. A simplified example MPD using ‘EventStream’ toindicate keeping statistics over a ‘Period’ is shown in Table 13.

TABLE 12 Example ‘Orientation stats keep’ scheme for ‘EventStream’.Element or Attribute Name Description EventStream @schemeIdUriurn:mpeg:dash:orientationstatskeep:2016

TABLE 13 Simplified example MPD signaling ‘orientation stats keep’ overa ‘Period’ using ‘EventStream’ <MPD>  <Period>   <!-- Keep statistics ofuser's head orientation over duration   of this period -->  <EventStream schemeIdUri=“urn:mpeg:dash:orientationstatskeep:   2016”/>   <AdaptationSet> . . . </AdaptationSet>   . . .  </Period> </MPD>

The ‘SupplementalProperty’ element may be used to indicate to the clientthat it is to start keeping statistics. An ‘orientation statskeep’scheme may specify this signal. The SupplementalProperty descriptor with@schemeIdUri equal to “um:mpeg:dash:orientationstatskeep:2016”, shown inTable 14, may indicate that keeping statistics may start at thebeginning and may stop at the end of the containing Period element. Asimplified example MPD using ‘SupplementalProperty’ to indicate keepingstatistics over a ‘Period’ is shown in Table 15.

TABLE 14 Example ‘Orientation stats keep’ scheme for‘SupplementalProperty’ Element or Attribute Name DescriptionSupplementalProperty @schemeIdUriurn:mpeg:dash:orientationstatskeep:2016

TABLE 15 Simplified sample MPD signaling ‘orientation stats keep’ over a‘Period’ using ‘SupplementalProperty’. <MPD>  <Period>   <AdaptationSet>   <!-- Keep stats of user's head orientation over the duration of thisperiod -->    <SupplementalPropertyschemeIdUri=“urn:mpeg:dash:orientationstatskeep:2016” />    . . .  </AdaptationSet>   . . .  </Period> </MPD>

An indication to start keeping statistics over part of a period may besent to a client. The indication may include one or more of, but notlimited to, an ‘InbandEventStream’ element, an ‘EventStream’ element,and/or the like. The ‘InbandEventStream’ and ‘SupplementalProperty’elements may be associated with ‘AdaptationSet’. In response toreceiving the indication to start keeping statistics, the client maykeep statistics during specific part of the period of video streamingsession.

Keeping statistics during a particular part of a period may be signaledto streaming clients using an ‘InbandEventStream’ element. The‘orientation stats keep’ scheme shown in Table 16 may be used to specifythis signal. The attributes in the ‘InbandEventStream’ and its child‘Event’ element may be used to indicate the part of the period wherestatistics may be kept. For the ‘orientation stats keep’ scheme, theattributes listed below may be used.

TABLE 16 Example use of ‘InbandEventStream’ for keeping statisticsduring part of a period Element or Attribute name Use DescriptionInbandEventStream May specify event stream @schemeIdUri Value may be:‘urn:mpeg:dash:orientationstatskeep:2016’ @timescale May specify thetimescale in units per seconds to be used for the derivation ofdifferent real-time duration values in the Event elements. Event 1 . . .N May specify one event. (e.g., as shown in Table 17). Legend: Forelements: <minOccurs> . . . <maxOccurs> (N = unbounded) Elements arebold; attributes are non-bold and preceded with an @.

TABLE 17 Example ‘Event’ semantics for ad viewports Element or Attributename Use Description Event May specify event stream @presentationTime MMay specify the presentation time of the event relative to the start ofthe Period @duration M May specify the presentation duration of theevent @id M May specify the id of the secondary content viewport thatcorresponds to this event

The value of the ‘@presentationTime’ and ‘@duration’ attributes inseconds may be obtained by dividing the value of these attributes by thevalue of the @timescale attribute. Multiple ‘Event’ elements may be usedto keep statistics multiple times over the duration of a ‘Period’.

TABLE 18 Simplified example MPD signaling ‘orientation stats keep’ overpart of a ‘Period’. <MPD>  <Period>   <AdaptationSet>    <!-- Startkeeping statistics user's head orientation 20 seconds after      thebeginning of the Period. Keeping statistics over 10 seconds. -->   <InbandEventStreamschemeIdUri=“urn:mpeg:dash:orientationstatskeep:2016” timescale=“1000”>    <Event presentationTime=“20000” duration=“10000” id=“viewport_1” />   </InbandEventStream>    ...   </AdaptationSet>   ...  </Period></MPD>

Keeping statistics during part of a period may be signaled to streamingclients using an ‘EventStream’ element. The ‘orientation stats keep’scheme shown in Table 19 may specify this signal, and attributes in the‘EventStream’ and its child ‘Event’ element may indicate the part of theperiod where keeping statistics is to take place. For the ‘orientationstats keep’ scheme, the attributes listed below are used.

TABLE 19 Example use of ‘EventStream’ for keeping statistics during partof a period Element or Attribute name Use Description EventStream Mayspecify event stream @schemeIdUri Value may be:‘urn:mpeg:dash:orientationstatskeep:2016’ @timescale May specify thetimescale in units per seconds to be used for the derivation ofdifferent real-time duration values in the Event elements. Event 1 . . .N May specify one event. For details see Table 20. Legend: For elements:<minOccurs> . . . <maxOccurs> (N = unbounded) Elements are bold;attributes are non-bold and preceded with an @.

TABLE 20 Example ‘Event’ semantics for secondary content viewportsElement or Attribute name Description Event May specify event stream@presentationTime May specify the presentation time of the eventrelative to the start of the Period @duration May specify thepresentation duration of the event @id May specify the id of thesecondary content viewport that corresponds to this event

The value of the ‘@presentationTime’ and ‘@duration’ attributes inseconds may be the division of the value of these attributes and thevalue of the @timescale attribute. Multiple ‘Event’ elements may be usedto keep statistics multiple times over the duration of a ‘Period’.

The ‘@id’ attribute may identify the secondary content viewportassociated with the ‘Event’. The ‘@id’ attribute may be used to matchmultiple ‘Event’ elements to multiple ‘AdaptationSet’ elements within a‘Period’. For example, when the same 360 degree video is available inmultiple different projections, ‘@id’ attribute may be used to matchmultiple ‘Event’ elements to multiple ‘AdaptationSet’ elements within a‘Period’. A simplified example MPD using ‘EventStream’ to indicatekeeping statistics over a ‘Period’ is shown in Table 21.

TABLE 21 Simplified example MPD signaling ‘orientation stats keep’ overpart of a ‘Period’ <MPD>  <Period>   <!-- Start keeping statisticsuser's head orientation 20 seconds after     the beginning of thePeriod. Keeping statistics over 10 seconds. -->   <EventStreamschemeIdUri=“urn:mpeg:dash:orientationstatskeep:2016” timescale=“1000”>   <Event presentationTime=“20000” duration=“10000” id=“viewport_1” />  </EventStream>   <AdaptationSet> ... </AdaptationSet>   ...  </Period></MPD>

Statistics of user's head orientation over an entire streaming sessionmay be kept. A ‘InbandEventStream’ element with @schemeIdUri equal to“um:mpeg:dash:orientationstatskeep:2016”, e.g., as shown in Table 10,within a ‘MPD’ element may be used to indicate that keeping statisticsmay start at the beginning and may stop at the end of the videostreaming session. A simplified MPD showing the use of this approach isshown in Table 22.

TABLE 22 Simplified example MPD signaling ‘orientation stats keep’ foran entire DASH streaming session <MPD>  <!-- Keep statistics of user'shead orientation over duration of this session -->  <InbandEventStreamschemeIdUri=“urn:mpeg:dash:orientationstatskeep:2016” />  <Period> ...</Period>  ...  <Period> ... </Period> </MPD>

The streaming client may determine secondary content viewport positionsbased on the received request. The areas in the 360 degree video thatare secondary content viewports may be specified via signaling.Secondary content viewports may be specified over an entire period, overpart of a period, or over an entire streaming session.

While the description herein is provided for 360 degree video inequirectangular projection, the concepts described here also apply ifother projections (e.g., cubemap) are used. Similarly, while thedescription herein is provided for a rectangular coordinate system, theconcepts described herein also apply if other coordinate systems (e.g.,spherical) are used.

Secondary content viewports may be specified over an entire ‘Period.’For example, secondary content viewports specified over an entire periodmay be signaled to streaming clients using a ‘SupplementalProperty’element. A secondary content viewport Relationship Description (AdVRD)scheme may specify the corresponding secondary content viewport spatialcoordinate relationships. For example, the SupplementalPropertydescriptor with (schemeIdUri equal to “um:mpeg:dash:adviewport:2d:2016”may be used to provide AdVRD associated with the containing‘AdaptationSet’, ‘Representation’ or ‘SubRepresentation’ element. The@value of the SupplementalProperty element using the AdVRD scheme mayinclude a comma separated list of values for secondary content viewportdescription parameters. An ‘AdaptationSet’, ‘Representation’ or‘SubRepresentation’ element may contain one or more AdVRDs to representone or more ad viewports. The properties of a secondary content viewportcan be described by the AdVRD parameters as shown in Table 23.

TABLE 23 Example secondary content viewport Relationship Description(AdVRD) Scheme. AdVRD scheme Description @schemeIdUriurn:mpeg:dash:adviewport:2d:2016 @value Comma-separated list of valuesfor the following fields: viewport_id, viewport_x, viewport_y,viewport_width, viewport_height, full_width and full_height viewport_idNon-negative integer in decimal representation providing a uniqueidentifier for secondary content viewport viewport_x non-negativeinteger in decimal representation expressing the horizontal position ofthe top-left comer of identified viewport. viewport_y non-negativeinteger in decimal representation expressing the vertical position ofthe top-left corner of the identified viewport viewport_widthnon-negative integer in decimal representation expressing the width ofidentified viewport viewport_height non-negative integer in decimalrepresentation expressing the height of identified viewport full_widthnon-negative integer in decimal representation expressing the width ofentire 360 degree video. If not present, the value may be inferred to bethe same as the value of viewport width. full_height non-negativeinteger in decimal representation expressing the height of entire 360degree video. If not present, the value may be inferred to be the sameas the value of viewport height.

The coordinates specified in the AdVRD may be consistent with the videotype, projection method and layout used for the 360 video in thecontaining ‘AdaptationSet’, ‘Representation’ or ‘SubRepresentation’element. As an example, a simplified MPD having one period with onesecondary content viewport is shown in Table 24.

TABLE 24 Simplified example MPD with one secondary content viewportdefined over an entire period. <MPD>  <!-- Period with a secondarycontent viewportsignalled using AdVRD -->  <Period id=“Main_content”>  <!-- 360 degree video with equirectangular with a secondary contentviewport-->   <AdaptationSet mimeType=“video/mp4” ... videoType=2>   <!-- Keep statistics of user's head orientation over duration of thisperiod -->    <InbandEventStreamschemeIdUri=“urn:mpeg:dash:orientationstatskeep:2016” />    <!--Definition of secondary content viewport -->    <SupplementalPropertyschemeIdUri=“urn:mpeg:dash:adviewport:2d:2016”    value=“1,1000,400,640,480,4096,2048”/>    <Representation @id=“0”width=“4096” height=“2048” bandwidth=“327597”>    <BaseURL>main_content.mp4</BaseURL>    </Representation>  </AdaptationSet>  </Period> </MPD>

FIG. 11 depicts an example viewport defined over several regions of acubemap projection. If the ‘@videoType’, ‘@projection’ and ‘@layout’combination result in a secondary content viewport laid out over severalregions of the 2D projected video, multiple AdVRDs may be defined toindicate the portions of the viewport in different regions, as shown inFIG. 11. The client may consolidate the statistics for the parts of theviewport prior to reporting orientation statistics to the server.

A secondary content viewport may be associated with part of a ‘Period.’Secondary content viewports specified over part of a period may besignaled to streaming clients using a ‘SupplementalProperty’ elementand/or the secondary content viewport Relationship Description (AdVRD)scheme in Table 23. A single period of, for example, a TV show, may becomposed of several scenes. Secondary content viewports may be definedover all of the scenes or some of these scenes.

As an example, a simplified MPD having one period with one secondarycontent viewport defined over part of a period is shown in Table 25. Inthe ‘Event’ element. ‘presentationTime’ may indicate the start time ofthe ad viewport, and ‘duration’ may indicate its duration. To obtain thetime in seconds. ‘presentationTime’ and ‘duration’ may be divided by thevalue of ‘timescale’ from the ‘InbandEventStream’ element.

TABLE 25 Simplified example MPD with a secondary content viewportdefined over part of a period <MPD>  <!-- Period with a secondarycontent viewportsignalled using AdVRD -->  <Period id=“Main_content”>  <!-- 360 degree video with equirectangular with a secondary contentviewport-->   <AdaptationSet mimeType=“video/mp4” ... videoType=2>   <!-- Start keeping statistics user's head orientation 20 secondsafter      beginning of Period. Keep statistics over 10 seconds forsecondary content viewport “1”. -->    <InbandEventStreamsehemeIdUri=“urn:mpeg:dash:orientationstatskeep:2016” timescale=“1000”>    <Event presentationTime=“20000” duration=“10000” id=“1” />   </InbandEventStream>    <!-- Definition of secondary content viewport-->    <SupplementalPropertyschemeIdUri=“urn:mpeg:dash:adviewport:2d:2016”    value=“1,1000,400,640,480,4096,2048”/>    <Representation @id=“0”width=“4096” height=“2048” bandwidth=“327597”>    <BaseURL>main_content.mp4</BaseURL>    </Representation>  </AdaptationSet>  </Period> </MPD>

A secondary content viewport may be associated with an entire streamingsession. Secondary content viewports specified over an entire streamingsession may be signaled to streaming clients using a‘SupplementalProperty’ element and the secondary content viewportRelationship Description (AdVRD) scheme in Table 23. For example, a‘SupplementalProperty’ element as defined in Table 25 at the MPD levelmay indicate secondary content viewports of interest. As an example, asimplified example MPD having one secondary content viewport definedover the entire streaming session is shown in Table 26.

TABLE 26 Simplified example MPD with one secondary content viewportdefined over an entire streaming session <MPD>  <!-- Keep statistics ofuser's head orientation over duration of streaming session --> <InbandEventStreamschemeIdUri=“urn:mpeg:dash:orientationstatskeep:2016” />  <!--Definition of secondary content viewport -->  <SupplementalPropertyschemeIdUri=“urn:mpeg:dash:adviewport:2d:2016”  value=“1,1000,400,640,480,4096,2048/>  <Period> ... </Period>  ... <Period> ... </Period> </MPD>

Orientation statistics over an entire streaming session may be used toidentify the most viewed areas of a 360 degree video. The client maygather data on a set of specified ad viewports, using the scheme shownas an example in Table 26.

Statistics of areas of the 360 degree video may be kept. FIG. 12 depictsan example 360 degree video frame partitioned in a 3×3 grid forgathering orientation statistics. The client may gather orientationstatistics over the areas of the 360 degree video. The client maypartition the 360 degree video frame into a grid (for example, 3×3 gridin FIG. 12) and may report orientation statistics over one or more ofthese areas at the end of the streaming session.

The grid size may be signaled to the client. For example, the‘InbandEventStream’ element with @schemeIdUri equal to“um:mpeg:dash:orientationstatskeep:2016” at the MPD level, may be usedto indicate the grid size, as shown in Table 27. The grid areas may benumbered in raster scan order from left to right and top to bottom,although other numbering orders may be used. The orientation statisticsreport may be sent to a server by the client. As an example, asimplified MPD for keeping orientations statistics over the entirestreaming session on the entire 360 degree video partitioned on a 3×3grid is shown in Table 28.

TABLE 27 Example ‘Orientation stats keep’ scheme indicating grid sizefor keeping statistics over areas of the 360 degree video Element orAttribute Name Description InbandEventStream @schemeIdUriurn:mpeg:dash:orientationstatskeep:2016 @value May specify the grid size(M × N) used to keep statistics over areas of the 360 degree video.

TABLE 28 Simplified example MPD for keeping orientation statistics overthe areas of a 360 degree video with a 3×3 grid <MPD>  <!-- Keepstatistics of user's head orientation over duration of streaming session-->  <InbandEventStreamschemeIdUri=“urn:mpeg:dash:orientationstatskeep:2016” value=“3×3”/> <Period> ... </Period>  ...  <Period> ... </Period> </MPD>

Statistics of some areas of the 360 degree video may be kept. The clientmay gather orientation statistics over some areas of the 360 degreevideo. The client may partition the 360 degree video frame into a grid(for example, 3×3 grid in FIG. 12) and may report orientation statisticsover a subset of these areas at the end of the streaming session.

The grid size may be signaled to the client. For example, the‘InbandEventStream’ element with @schemeIdUri equal to“um:mpeg:dash:orientationstatskeep:2016” at the MPD level, may be usedto indicate the grid size, as shown in Table 29. An attribute (@ areasmay indicate a list of area numbers over which statistics are kept, asshown in Table 29. The grid areas may be numbered in raster scan orderfrom left to right and top to bottom, although other numbering ordersmay be used. The orientation statistics report may be sent to a serverby the client. As an example, a simplified MPD for keeping orientationsstatistics over the entire streaming session on areas 4, 5 and 6 of a360 degree video partitioned on a 3×3 grid is shown in Table 30.

TABLE 29 ‘Orientation stats keep’ scheme indicating grid size forkeeping statistics over some areas of the 360 degree video. Element orAttribute Name Description InbandEventStream @schemeIdUriurn:mpeg:dash:orientationstatskeep:2016 @value May specify the grid size(M × N) used to keep statistics over some areas of the 360 degree video.@areas May specify the list of areas over which statistics are kept.List of areas (identified by numbers between 1 and M × N) iscomma-separated. Invalid area numbers may be ignored.

TABLE 30 Simplified example MPD with for keeping orientation statisticsover an entire streaming session with a 3×3 grid. <MPD>  <!-- Keepstatistics of user's head orientation over duration of streaming session-->  <InbandEventStreamschemeIdUri=“urn:mpeg:dash:orientationstatskeep:2016” value=“3×3”  areas=“4,5,6” />  <Period> . . . </Period.>  . . .  <Period> . . .</Period> </MPD>

While the description of this section assumes that position of secondarycontent viewports is static, the concepts described herein also apply ifthe streaming system supports a method for identifying and trackingmoving objects in 360 degree video. For example, if a moving car isshown in a scene, this moving object may be defined as a secondarycontent viewport and may be signaled to the client as such.

A streaming client may determine which ad to show based on a list of adsin the MPD and the relation between the ads and ad viewports. The listof secondary contents may be specified within a single period. Asecondary content in the list may be associated with a secondary contentviewpoint. The relation between secondary contents and their respectivesecondary content viewports may be established via secondary contentviewport ids.

Secondary content(s) may be selected based on the tracked secondarycontent viewport parameter and/or the relation between secondarycontents and their respective secondary content viewports. For example,the ‘EssentialProperty’ descriptor with @schemeIdUri equal to“um:mpeg:dash:adselection:2016” may specify the relation between thesecondary content viewport and the secondary content in the containing‘AdaptationSet’, ‘Representation’ or ‘SubRepresentation’ element. In alist of secondary contents, the ‘EssentialProperty’ descriptor may usethe same @id, indicating that the processing of one of the descriptorswith an identical value may be sufficient. For example, a secondarycontent may be selected from the list. The relation to secondary contentviewports may be established by setting the (value to the secondarycontent viewport id of the corresponding secondary content viewport. Forthe ‘secondary content selection’ scheme, the attributes are listed inTable 31.

TABLE 31 Example semantics of the elements and attributes of the‘secondary content selection’ scheme Element or Attribute NameDescription EssentialProperty @schemeIdUri um:mpeg:dash:adselection:2016@id May specify an identifier for the descriptor. Identical values forthis attribute may be used for secondary contents in the same list,indicating that the selection of one of the descriptors with anidentical value may be sufficient. @value Corresponds to the@viewport_id attribute of the secondary content viewport thatcorresponds to this descriptor. The special value ‘9999’ may be reservedfor the default ad, that is, the ad that may be used if none of theother ads are selected.

Whether to provide ad periods with secondary content viewportinformation to a client may be conditioned on the client's capabilities.For example, ad periods with secondary content viewport information maybe given to streaming clients that support secondary content viewports.If the scheme of the ‘EssentialProperty’ descriptor is not recognized bythe DASH client, the client may not receive ad periods with secondarycontent viewport information.

As an example, a simplified MPD having one period with two ad viewports,followed by a period with a list of ads that correspond to these adviewports, is shown in Table 32. If multiple secondary content viewportsare defined, the same ad may be shown for multiple viewports. A defaultad may be listed, for example, in case the user's head orientation doesnot fall within any of the ad viewports.

TABLE 32 Simplified example MPD showing ad selection based on adviewports. <MPD>  <!—Main content period with secondary contentviewports signalled using AdVRD -->  <Period id=“Main_content”>   <!--360 degree video with equirectangular with two secondary contentviewports -->   <AdaptationSet mimeType=“video/mp4” ... videoType=2>   <!-- Keep statistics of user's head orientation over duration of thisperiod -->    <InbandEventStreamschemeIdUri=“urn:mpeg:dash:orientationstatskeep:2016” />    <!--Definition of secondary content viewport with id=1 -->   <SupplementalProperty schemeIdUri=“urn:mpeg:dash:adviewport:2d:2016”    value=“1,1000,400,640,480,4096,2048/>    <!-- Definition ofsecondary content viewport with id=2 -->    <SupplementalPropertyschemeIdUri=“urn:mpeg:dash:adviewport:2d:2016”    value=“2,500,800,640,480,4096,2048”/>    <Representation @id=“0”width=“4096” height=“2048” bandwidth=“327597”>    <BaseURL>main_content.mp4</BaseURL>    </Representation>  </AdaptationSet>  </Period>  <!-- Ad period with list of ads fromwhich one is selected based on secondary content viewport -->  <Periodid=“Ad_period”>   <!-- List of ads -->   <AdaptationSetmimeType=“video/mp4” ... videoType=2>    <!-- Show this ad if userattention was on viewport with id=1 (set value=1) -->   <EssentialProperty schemeIdUri=“urn:mpeg:dash:adselection:2016”id=“Ad_1” value=“1”/>    <Representation @id=“0” width=“4096”height=“2048” bandwidth=“327597”>    <BaseURL>ad_related_to_viewport_1.mp4</BaseURL>    </Representation>  </AdaptationSet>   <AdaptationSet mimeType=“video/mp4” ...videoType=2>    <!-- Show this ad if user attention was on viewport withid=2 (set value=2) -->    <EssentialPropertyschemeIdUri=“urn:mpeg:dash:adselection:2016” id=“Ad_2” value=“2”/>   <Representation @id=“0” width=“4096” height=“2048”bandwidth=“327597”>     <BaseURL>ad_related_to_viewport_2.mp4</BaseURL>   </Representation>   </AdaptationSet>   <AdaptationSetmimeType=“video/mp4” ... videoType=2>    <!-- This is default ad if userattention was not on any viewport (set value=9999)-->   <EssentialProperty schemeIdUri=“urn:mpeg:dash:adselection:2016”id=“Ad_default” value=“9999”/>    <Representation @id=“0” width=“4096”height=“2048” bandwidth=“327597”>     <BaseURL>default_ad.mp4</BaseURL>   </Representation>   </AdaptationSet>  </Period> </MPD>

The list of secondary content viewports may be different per user, andmay vary between views of the same content by the same user. Forexample, multiple ads may be shown by using multiple ad periods, eachwith corresponding secondary content viewport information.

An indication that reporting orientation statistics is desired may besent to streaming clients using an ‘InbandEventStream’ element. The‘orientation stats report’ scheme shown in Table 33 may be used tospecify this signal.

TABLE 33 ‘Orientation stats report’ scheme for ‘InbandEventStream’Element or Attribute Name Description InbandEventStream @schemeIdUriurn:mpeg:dash:orientationstatsreport:2016 @reportServer:href URL ofserver where statistics may be reported.

The signal to indicate that reporting orientation statistics is desiredmay follow the signal to start keeping statistics within a ‘Period’ (orpart of a ‘Period’) or over the entire streaming session, as describedherein.

A simplified example MPD using ‘InbandEventStream’ to indicate thatreporting statistics gathered over a ‘Period’ is desired is shown inTable 34.

TABLE 34 Simplified example MPD signaling ‘orientation stats report’over a ‘Period’ <MPD>  <Period>   <AdaptationSet>    <!-- Keepstatistics of user's head orientation over duration of this period -->   <InbandEventStreamschemeIdUri=“urn:mpeg:dash:orientationstatskeep:2016” />    <!-- Reportorientation statistics gathered over duration of this period -->   <InbandEventStreamschemeIdUri=“urn:mpeg:dash:orientationstatsreport:2016”reportServer=“http://report.server.com/” />    ...   </AdaptationSet>  ...  </Period> </MPD>

A simplified example MPD using ‘InbandEventStream’ to indicate thatreporting statistics gathered over a streaming session is desired isshown in Table 35.

TABLE 35 Simplified example MPD signaling ‘orientation stats report’over a streaming session <MPD>  <!-- Keep statistics of user's headorientation over duration of this session -->  <InbandEventStreamschemeIdUri=“urn:mpeg:dash:orientationstatskeep:2016” />  <!-- Reportorientation statistics at the end of this session --> <InbandEventStreamschemeIdUri=“urn:mpeg:dash:orientationstatsreport:2016”reportServer=“https://report.server.com/” />  <Period> ... </Period> ...  <Period> ... </Period> </MPD>

The timing for reporting statistics may be pre-determined or dynamicallysignaled. For example, reporting may take place at the end of thecontaining Period element, may take place at the end of the streamingsession, and/or the streaming client may save multiple report for batchuploading (e.g., nightly, weekly).

Upon determining which secondary content has been selected based onsecondary content viewport information, the secondary content may befetched from the content server and presented to the viewer. A 360degree secondary content may be presented in a 360 degree environment.3D and/or rectilinear (2D) secondary contents may be presented for 360degree video content.

For example, a 2D secondary content may be presented as a 2D videowithin the 360 degree video environment. For example, HMDs for 360degree video may provide support for displaying content having multiplelayers. For example, a layer may be the 360 degree video content andanother layer may be a head-up display (HUD) layer with informationabout the content being shown, a navigation menu, or a 2D secondarycontent. The 2D secondary content may be placed in a fixed positionwithin the 360 degree video content such that the 2D secondary contentmay move in response to changes in user orientation. The 2D secondarycontent may be placed in a fixed position relative to the user'sorientation (e.g. “headlocked”), such that changes in user orientationmay not move the 2D secondary content, and such content maybe rooted inthe same position within the user's view. In some embodiments, the 2Dsecondary content may blend into the other content in the 360 degreevideo. For example, an ad for a car may add the image of the car intothe 360 video in a place where a car might naturally park within thescene. This effect may be realized by jointly designing the main 360video content and the ad content. Specific secondary content insertionlocations in which specific ads are designed to blend in may be receivedvia signaling.

The 2D secondary content may be shown in the user's view. It may coverthe viewport shown to the user. It may be scaled to appear smaller orlarger within the viewport. The 2D secondary content may be placed indifferent positions within the viewport. For example, the 2D secondarycontent may be placed at the top, bottom, left, and/or right of theviewport. The 2D secondary content may be placed outside of the currentviewport, such that the user may not be startled or may not have arealistic experience interrupted by new content suddenly appearing inthe field of view. In this case, the 2D secondary content may be placedin a direction that the user is likely to look in the near future.Visual or audio hints may be used to prompt the user to look in thedirection where the 2D secondary content (e.g. an ad) is placed.

FIG. 13 shows an example screenshot of a 360 degree video with an ad. Ascan be seen, the secondary content may be placed in the user's view, butit does not cover the entire viewport.

Secondary content insertion location may be received. DASH signaling maybe used to indicate to the streaming client information about secondarycontent insertion location (e.g., where the secondary content may beinserted). For example, the size and position of a 2D ad within a 360degree environment may be signaled to the streaming client. TheSupplementalProperty descriptor with @schemeIdUri equal to“um:mpeg:dash:adposition:2d:2016” may indicate 2D ad informationassociated with the containing ‘AdaptationSet’, ‘Representation’ or‘SubRepresentation’ element.

The @(value of the SupplementalProperty element using the ad positionscheme may include a comma separated list of values for secondarycontent insertion position parameters. An ‘AdaptationSet’,‘Representation’ or ‘SubRepresentation’ element may contain none or asecondary content position parameter for an ad period. Exampleproperties of an ad position can be described by the parameters shown inTable 36.

TABLE 36 Secondary content position scheme Secondary content positionscheme Description @schemeIdUri urn:mpeg:dash:adposition:2d:2016 @valueComma-separated list of values for the following fields: position_id,scale, position and position_y position_id Non-negative integer indecimal representation providing a unique identifier for secondarycontent insertion position position_x non-negative integer in decimalrepresentation expressing the horizontal position of the top-left comerof secondary content within the 360 degree environment. position_ynon-negative integer in decimal representation expressing the verticalposition of the top-left corner of secondary content within the 360degree environment. notify Boolean (‘true’, ‘false’) expressing whetherattention of the user is called if the secondary content does not appearin the viewport being shown to the user. Default value may be ‘true’.

For example, the “notify” parameter may include a Boolean true/falseexpressing whether notification should be presented to the user in casethe secondary content is displayed outside of the user's current view.For example, if the value is set to true, the user may be alerted insome way (e.g. a sound may be played or a graphical hint may bedisplayed to alert the user that new content has appeared, or adirectional indicator may be displayed to point the user towards theout-of-view secondary content).

In an example, the secondary content is to be presented to the viewerregardless of where the viewer is looking at that time. An indication,such as parameter “notify” shown in Table 36, may indicate that thesecondary content is to be presented to the viewer regardless of wherethe viewer is looking at that time. For example, if the “notify”parameter in Table 36 is set to “true”, the streaming client may ignorethe values ‘position_x’ and ‘position_y’ and show the secondary contentin the viewport that the user is currently looking at (e.g., the currentviewport).

The coordinates specified in the secondary content position scheme maybe consistent with the video type, projection method and layout used forthe 360 video in the containing ‘AdaptationSet’, ‘Representation’ or‘SubRepresentation’ element.

As an example, a simplified example MPD having an ad period with adposition information is shown in Table 37. In this example, the 2D ad isplaced at the (920,640) position within the 360 degree environment. Theuser may not be notified of the ad, if the ad is shown outside of theviewport being shown to the user. In this case, the 360 degree video maybe paused, and the ad may be shown outside of the view of the user. Theaudio of the secondary content may play, prompting the user to lookaround in the 360 degree environment and see the secondary content.

TABLE 37 Simplified example MPD of one ad period with ad positioninformation <MPD>  <!-- Ad period with list of ads from which one isselected based on secondary content viewport -->  <Periodid=“Ad_period”>   <!-- List of ads -->   <AdaptationSetmimeType=“video/mp4” ... videoType=2>    <!-- Show this ad if userattention was on viewport with id=1 (set value=1) -->   <EssentialProperty schemeIdUri=“urn:mpeg:dash:adselection:2016”id=“Ad_1” value=“1”/>    <SupplementalPropertyschemeIdUri=“urn:mpeg:dash:adposition:2d:2016”value=“1,920,640,50,false”/>    <Representation @id=“0” width=“4096”height=“2048” bandwidth=“327597”>    <BaseURL>ad_related_to_viewport_1.mp4</BaseURL>    </Representation>  </AdaptationSet>  </Period> </MPD>

A streaming client may keep statistics about users' interest onsecondary content viewports during a DASH video streaming session. FIG.14 shows an example streaming client keeping orientation statistics,enabling system to make decisions using secondary content viewports. Asshown, statistics may be kept over a Period, over part of a Periodand/or over a streaming session.

The viewport shown to the user may be determined based on headorientation information. Statistics about orientation may be trackedduring a streaming session, and computed to support ad viewports. Theregions within the 360 degree video frame that the user is watching maybe captured. Orientation at a particular instant in time may be used todetermine whether the secondary content viewport is shown (e.g., whetherthe viewer is paying attention to the secondary content viewport).Statistics about user interest in secondary content viewports may betracked for secondary content viewports and/or for areas of a 360 degreevideo. Timing of a decoded frame may be used to determine whether itfalls within the time interval over which the secondary content viewportis defined. Whether the user's head orientation at a given point in timefalls within a secondary content viewport may be determined.

One or more secondary content viewport parameters may be tracked.Secondary content viewport parameters may include secondary contentviewport statistics. FIG. 15 shows an example of keeping statistics oforientation for use with ad viewports. At 1501, secondary contentviewport information may be obtained upon parsing information (e.g. a‘Period’) in the MPD. Secondary content viewport statistics may be keptusing simple counters, with one counter per ad viewport. Counters may beassociated to a viewport using the ‘viewport_id’ listed in the MPD, andmay be set to zero at the beginning of the algorithm. At 1502, when neworientation information is obtained, it may be compared against thesecondary content viewports listed for the corresponding period in theMPD. Some secondary content viewports may be defined for part of aperiod. The client may determine whether a secondary content viewport is“active”. For secondary content viewports that are defined over part ofa ‘Period’, the presentation timestamp of the decoded frame may bewithin the time over which the secondary content viewport is defined, asspecified by the ‘InbandEventStream’ element. For example, at 1503, thepresentation time (PT) of the decoded video frame may be comparedagainst the time interval over which the secondary content viewport isdefined. If the secondary content viewport is defined over the entire‘Period’, this comparison may be skipped. At 1504, if the currentorientation falls within one of the viewports, the corresponding countermay be increased accordingly (e.g., by one) at 1505. Otherwise, thecounter may not be increased. The counter may track (e.g. approximately)the amount of time that a user has viewed a particular secondary contentviewport. The largest counter may indicate the secondary contentviewport that has been seen by the user the most. This result may beused to select a secondary content, at 1506. For example, the adselection decision may be made by the client and/or by an ad decisionserver. At 1507, orientation statistics may be reported by the clientupon completion of the streaming session.

FIG. 16 shows an example of keeping statistics of orientation over areasof a 360 degree video. Certain processes may be similar to thosedescribed in FIG. 15. Grid area size may be obtained from the MPD (e.g.,as indicated by the ‘InbandEventStream’ element at the MPD level). At1601, a counter may be defined for each grid area. A different set ofcounters may be used for each period. A set of counters may beidentified by the period's ‘@id’ attribute. At 1602, the orientationinformation may be obtained, and a grid area corresponding to theorientation information may be mapped or determined at 1603. Forexample, the grid area which best aligns with the center of the user'sview may be selected at 1603, and its corresponding counter may beincreased accordingly (e.g., by one) at 1604. At the end of thestreaming session, statistics information (e.g., one counter per gridarea) may be reported (e.g., to a server) at 1605. Although not shown inFIG. 16, additional steps that utilize the statistics information (e.g.at the client and/or at an ad decision server) to select secondarycontent (e.g. an ad) and to display such content to the user may beperformed (see for example 1506 of FIG. 15).

Video timing information may be matched to a secondary content viewportdefinition. The presentation time of a decoded frame may be matched tothe definition of the secondary content viewport in the MPD. Thematching may be performed for secondary content viewports that aredefined over part of a ‘Period’, as secondary content viewports are not“active” for frames in a ‘Period’. If a secondary content viewport isdefined over a complete ‘Period’, the streaming client may determinethat the frames that belong to the ‘Period’ are “active”.

In a DASH streaming system, the video decoder module may receive asinput the bitstream that is contained in downloaded segments, and mayprovide as output decoded frames in the order to be displayed. Thedecoder may perform video frame reordering as part of inter-frameprediction. During decoding of MP4 file segments, the presentation time(PT) of a decoded frame may be derived from the frame rate of the videoand from the frame number, assuming that the time at the beginning ofthe ‘Period’ is zero. For example, the PT of the 120th frame of a videoat 30 frames per second is: 120×(1/30)=4 seconds. Based on the MPD, thestreaming clients may determine the time at which the ‘Period’ starts,the time at which a secondary content viewport becomes active (from the@presentation attribute of the ‘Event’ element), and for how long thesecondary content viewport is active (from the @duration attribute ofthe ‘Event’ element). The streaming client can determine whether thesecondary content viewport is active for a decoded frame based on the PTof the decoded frame.

Whether current orientation falls within secondary content viewport maybe determined. 360 degree video players may provide APIs for obtaininguser's head orientation. Applications that support 360 degree video mayprovide APIs for determining viewport during playback. The orientationinformation may include spherical coordinates [θ, ϕ], yaw/roll/pitch;[x,y,z], and the like for denoting a point in a sphere where the user islooking at.

As shown in FIG. 15, at 1504, the point in a sphere may be mapped from apoint in the sphere into a point in the projection of the 360 degreevideo frame over which secondary content viewports are defined, as shownin FIG. 17. This mapping may be performed depending on the projectionmethod (e.g., equirectangular, cubemap, pyramid) used in the 360 degreevideo. For example, if spherical coordinates are used, equations (1) and(2) may be used to map point P with coordinates (θ, ϕ) on a sphere to apoint P with coordinate (u, v) on a 2D plane. Whether the point iswithin a secondary content viewport may be performed, e.g., by comparingthe 2D coordinates of the user's head orientation against the 2Dcoordinates of the secondary content viewport definition.

u=ϕ/(2*pi)+0.5  (1)

v=0.5−θ/(pi)  (2)

As shown in FIG. 16, at 1603, the point P in a sphere is mapped into apoint in the projection of the 360 degree video frame over which a gridis defined, as shown in FIG. 18. This mapping depends on the projectionmethod (e.g., equirectangular, cubemap, pyramid) used in the 360 degreevideo. Whether the point is within a grid area may per determined bycomparing the 2D coordinates of the user's head orientation against the2D coordinates of the grids.

Viewing orientation may be classified based on secondary contentviewports (e.g., 1503-1505 shown in FIG. 15), and such classificationand/or counter input may be performed less than once per frame. Forexample, classification may be performed twice per second, thus reducingcomplexity by 15× or more, depending on the frame rate (e.g., frames persecond) of the 360 degree video, see Table 38. A computationalcomplexity level may be determined and adjusted based on streamingclient's setting and/or capabilities. Determining current orientationmay continue as part of ordinary 360 degree video playback.

TABLE 38 Reduction in computational requirements using less frequentsecondary content viewport classification. Secondary content Secondarycontent Video frame viewport classified viewport classified rate everyframe twice per second 30 fps Baseline 15x reduction 60 fps Baseline 30xreduction

Secondary content insertion may be determined based on secondary contentviewport parameter. The tracked secondary content viewport parametersinformation may be reported to a server. FIG. 19 shows an example ofviewing statistics on secondary content viewports that may be used todetermine which secondary content may be presented in a streamingsession.

After the streaming client keeps statistics of user's head orientationover a ‘Period’, part of a ‘Period’, or over an entire streamingsession, it may determine whether user's attention was on one or moresecondary content viewports. This information may be used to select asecondary content(s) relevant to the secondary content viewport(s). Thesecondary content may be selected by the streaming client and/or by asecondary content decision server. Orientation information may bereported by the streaming client to a server.

Secondary content decision may be performed at a streaming client basedon secondary content viewport statistics, for example, at 1506 shown inFIG. 15. For example, which ad to show may be determined. Secondarycontent viewport definitions and a list of possible secondary contentsto show based on secondary content viewport information/statistics maybe listed in the MPD as shown in Table 32. FIG. 20 shows an example ofselecting secondary content based on secondary content viewports.

For example, after a period with secondary content viewport(s) completesplayback, the counter(s) that correspond to secondary contentviewport(s) may be sorted in descending order. The ‘@viewport_id’attribute of the secondary content viewport with the largest counter maybe identified. For example, in case of a tie, either one of thesecondary content viewports may be selected. If the counters are zero, a‘@viewport_id’ equal to ‘9999’ may be used. In ad ‘Period’ in MPD, the‘AdaptationSet’ that has a ‘@value’ in ‘EssentialProperty’ (e.g.,representing the viewport_id of the ad that corresponds to thisdescriptor) equal to ‘@viewport_id’ may be searched. If an‘AdaptationSet’ is found, then it is selected and the ad may be playedback. If no ‘AdaptationSet’ is found, the default secondary content(@value equal to ‘9999’) may be selected and may be played back. If nodefault secondary content is listed, then the ad period may be skipped.

Secondary content decision may be performed at a secondary contentdecision server. For example, information about user's head orientationmay be sent to the secondary content decision server. This informationmay enable the secondary content decision server to determine whichsecondary content to show. Information may be sent by the client to thesecondary content decision server as part of the ‘xlink’ URL(server-based architecture) or as part of the secondary content MPD URL(app-based architecture).

Content servers may know secondary content viewport information(definition and secondary content viewport id(s)), such that an MPD withthis information may be created and provided to clients (contentserver). Secondary content servers may receive secondary contentviewport id(s), and may choose a secondary content based on thesecondary content viewport reported by the client.

Remote Periods may be used for ad insertion by using ‘xlink’ elementswith a URL from which an ad period is obtained by the client. For usewith secondary content viewports, a query parameter adviewport=x may beappended to the XLink URL used to get the ad period from the ad decisionservice, where ‘x’ may represent the id of the secondary contentviewport that was more interesting to the user. For example, considerthe 360 degree video with two secondary content viewports and a remoteperiod shown in Table 39. If the secondary content viewport withviewport_id=2 is seen by the user the most, the requesting client mayappend the query parameter ‘adviewport=2’ to the XLink URL as shownbelow:

https://adserv.com/avail.mpd?scte35-time=PT600.6S&scte35-cue=DAIAAAAAAAAAAAQAAZ_I0VniQAQAgBDVUVJQAAAAH+cAAAAAA%3D%3D &adviewport=2.

For example, the secondary content decision server may select an ad‘Period’ by using one or more of: information in the secondary contentviewport query (in this example, the query parameter ‘adviewport=2’),information about the relation between secondary content viewports andads, shown in Table 40, or client information passed as additionalparameters in the request or included in the request context (e.g., HTTPcookies or device ad identifiers). For example, there may be multiplelists of ad periods (Table 40) for different viewers or devices.

TABLE 39 Simplified sample MPD of a 360 degree video with secondarycontent insertion using remote periods. <MPD>  <!-- Main content periodwith secondary content viewports signalled using AdVRD -->  <Periodid=“Main_content”>   <!-- 360 degree video with equirectangular with twosecondary content viewports -->   <AdaptationSet mimeType=“video/mp4”... videoType=2>    <!-- Keep statistics of user's head orientation overduration of this period -->    <InbandEventStreamschemeIdUri=“urn:mpeg:dash:orientationstatskeep:2016” />    <!--Definition of secondary content viewport with id=1 -->   <SupplementalProperty schemeIdUri=“urn:mpeg:dash:adviewport:2d:2016”    value=“1,1000,400,640,480,4096,2048/>    <!-- Definition ofsecondary content viewport with id=2 -->    <SupplementalPropertyschemeIdUri=“urn:mpeg:dash:adviewport:2d:2016”    value=“2,500,800,640,480,4096,2048”/>    <Representation @id“0”width=“4096” height=“2048” bandwidth=“327597”>    <BaseURL>main_content.mp4</BaseURL>    </Representation>  </AdaptationSet>  </Period>  <!-- Ad is a remote period. Passingbase64url-coded SCTE 35 to XLink resolver -->  <Periodduration=“PT60.6S” id=“ad_break_1”  xlink:href=“https://adserv.com/avail.mpd?scte35-time=PT600.6S&scte35-cue=DAIAAAAAAAAAAAQAAZ_I0VniQAQAgBDVUVJQAAAAH+3AAAAAA%3D%3 D”  xlink:actuate=“onRequest” >  </Period> </MPD>

TABLE 40 List of ad periods from which one may be selected by secondarycontent decision server based on secondary content viewport information<!-- Ad for secondary content viewport with id=1 --> <Periodid=“Ad_period”>  <AdaptationSet mimeType=“video/mp4” ... videoType=2>  <!-- Show this ad if user attention was on viewport with id=1 (setvalue=1) -->   <Representation @id=“0” width=“4096” height=“2048”bandwidth=“327597”>    <BaseURL>ad_related_to_viewport_1.mp4</BaseURL>  </Representation>  </AdaptationSet> </Period> <!-- Ad for secondarycontent viewport with id=2 --> <Period id=“Ad_period”>  <AdaptationSetmimeType=“video/mp4” ... videoType=2>   <!-- Show this ad if userattention was on viewport with id=2 (set value=2) -->   <Representation@id=“0” width=“4096” height=“2048” bandwidth=“327597”>   <BaseURL>ad_related_to_viewport_2.mp4</BaseURL>   </Representation> </AdaptationSet> </Period> <!-- Ad used if user attention was not onany viewport --> <Period id=“Ad_period”>  <AdaptationSetmimeType=“video/mp4” ... videoType=2>   <!-- This is default ad if userattention was not on any viewport (set value =9999)-->   <Representation@id=“0” width=“4096” height=“2048” bandwidth=“327597”>   <BaseURL>default_ad.mp4</BaseURL>   </Representation> </AdaptationSet> </Period>

FIG. 21 shows an example call flow of a DASH presentation with secondarycontent insertion using remote periods and secondary content viewports.

FIG. 22 shows an example of selecting a secondary content usingsecondary content viewports. The secondary content decision server mayselect secondary content(s) based on the request received from theclient containing the secondary content viewport that was seen the mostby the user. Using the main content identifier (e.g., SCTE 35 cuemessage or DASH ‘AssetIdentifier’), a list of secondary content may beselected. The selection of the list may be made using client informationpassed as parameters in the HTTP request or using request context (e.g.,HTTP cookies or device ad identifier). The secondary content decisionserver may search the list for a matching secondary content viewport. Ifa match is found, the corresponding secondary content period may be sentback to the client. Otherwise, a default secondary content may be sentto the client or the ad break may be not taken (resulting in a Periodwith zero length). For simplicity, FIG. 22 shows the selection of asingle secondary content. If multiple secondary contents are to beshown, the steps may be executed multiple times, or the list held by thesecondary content decision server may have multiple secondary contentperiods for the same secondary content viewport.

For example, an in-client app may obtain an MPD with an ad ‘Period’ (orperiods) from the ad decision server by sending an HTTP request with cueinformation (which may uniquely identify the main content). A URL may beused to send an HTTP request from the app to the ad decision server. Foruse with secondary content viewports, a query parameter adviewport=x (ora similar parameter that may identify a secondary content viewport) maybe appended to the HTTP request sent to the ad decision server, where‘x’ may indicate the id of the secondary content viewport that was mostwatched by the user. The ad decision server may decide which ad toselect based on the secondary content viewport information, for example,as shown in FIG. 22. FIG. 22 shows an example of selecting an ad usingsecondary content viewports. FIG. 23 shows an example call flow of aDASH presentation with app-based secondary content insertion usingsecondary content viewports.

A streaming client may report statistics about orientation informationto a server, for example, at 1507 as shown in FIG. 15 and at 1605 inFIG. 16. Orientation information may be reported to a server foridentifying the secondary content viewports or the areas of 360 degreevideo that are most watched by users. Orientation statistics may be keptover a set of secondary content viewports or over areas of the 360degree video.

For reporting statistics for a set of secondary content viewportsdefined over a period (or part of a period), the client may have one setof counters per period. Statistics for periods for which no secondarycontent viewports are defined may be empty. The client may report the idof content for which statistics are being reported, the period overwhich the secondary content viewport is defined, the secondary contentviewport identifier, and/or the value of the counter.

For reporting statistics for a set of secondary content viewportsdefined over an entire streaming session, the client has one set ofcounters per period, because statistics are kept on a per-period basis(e.g., at 1501 as shown in FIG. 15). The streaming client may report theid of content for which statistics are being reported, the period overwhich the counter is reported, the secondary content viewportidentifier, and/or the value of the counter.

For reporting statistics over areas of a 360 degree video, the streamingclient may use a set of counters per period. Statistics may be kept on aper-period basis (e.g., at 1601 of FIG. 16). There may be M×N counters,a counter for an area on the grid defined in the MPD. The streamingclient may report, for a period, a counter per grid area. The number ofperiods in the session may be included as a separate element in thereport. The streaming client's report may have K×M×N counters for asession with ‘K’ periods.

The format of the report can be freeform, if streaming clients andserver agree to use the same format. For example, the information may bereported as an XML form, such as the one shown in Table 41. Otherformats may be used (e.g., comma separated values in a text file, etc.).

TABLE 41 Example of a report of orientation information over areas of a360 degree video for a streaming session. <?xml version=“1.0”encoding=“UTF-8”?> <REPORT> <!-- Id of content for which statistics arebeing reported --> <ContentId value=“xyz”> <!-- Grid size --> <GridSizevalue=“M×N”> <!-- Number of Periods in report --> <Periods value=“K”><!-- Statistics for Period 1 --> <Period>  <!-- One value per grid area.Total of M×N counters. -->  <Count value=“...”>  ... </Period> <!--Statistics for Period K --> <Period>  <!-- One value per grid area.Total of M×N counters. -->  <Count value=“...”>  ... </Period> </REPORT>

The streaming client may send the report to the server in a manner thatfits the application. For example, via an HTTP POST request, by loggingthe information in the device and periodically uploading it to theserver, and/or by passing the report another application in the clientdevice (e.g., reporting app).

Secondary content viewports may be used beyond ad insertion. Secondarycontent viewports may be defined for areas in a 360 degree videospecifically created for E-Commerce or in any 360 degree video whereproduct placement may be. Users may look at these products during videoplayback. Information such as views of the product, more detailed viewsof the product, zoomed images, detailed information, clickable links, orthe like may be presented to the user based on their interest on some ofthese products.

E-commerce using secondary content viewports may be provided. FIG. 24 isan example block diagram of e-commerce application with secondarycontent viewports. A 360 degree video (for example, one with productplacement or with objects that may attract the attention of the user)may be annotated using secondary content viewports as described herein.Annotating may include using secondary content viewports in the MPD toidentify the areas where the products or objects may be found on the 360degree video scene during the streaming session. The video may be madeavailable to a streaming server to be used with streaming clientsrunning an e-commerce application that may identify when the user'sinterest is on the areas of the 360 degree video scene that areidentified with secondary content viewports such as ad viewports.

Whether to pause the primary content of the 360 degree video to showsecondary content may be determined based on the secondary contentviewport parameter(s). For example, the primary content may be paused,and a secondary content associated with the secondary content viewportmay be shown during an intermission of playback of the main content. Forexample, whether a viewer's attention has been on a secondary contentviewport for more than a threshold period may be determined. Based on adetermination that the secondary content viewport has been viewed for aperiod of time, the primary content may be paused, and a secondarycontent associated with the secondary content viewport may be shown. Forexample, whether a viewer has taken an action to request additionalinformation on the content shown in the secondary content viewport maybe determined. Based on a determination that the viewer has requestedadditional information, the primary content may be paused and asecondary content associated with the secondary content viewport may beshown.

FIG. 25 shows an example e-commerce application using secondary contentviewports. The c-commerce application with secondary content viewportsmay run on a 360 degree video streaming client. The application maystart a streaming session by requesting the MPD of a 360 degree videoannotated with secondary content viewports from an HTTP server. Theapplication may parse the MPD to get detailed information aboutsecondary content viewports (e.g., timing and location information ofsecondary content viewports during the streaming session). When asecondary content viewport is active over a 360 degree video scene, thesecondary content viewport area may be highlighted, for example byoverlaying a rectangle, thus enabling the user to know that additionalinformation about a product is available if he/she is interested, at2501. FIG. 26 shows an example 360 degree video scene with areashighlighted by an e-commerce application.

The application may keep statistics about the user's head orientation.The application may determine whether user's attention is on a secondarycontent viewport based on the head orientation statistics.

As shown in FIG. 25, at 2502, the application may determine whether theuser's attention has been on a secondary content viewport over apre-determined period of time. The amount of time may be signaled on theMPD as part of the secondary content viewport information, or it may bepart of the configuration of the e-commerce application. The applicationmay receive a user indication that may indicate his/her desire to getadditional information on the item or product shown on the secondarycontent viewport by other means. For example, the indication may includeone or more of a tap on a touchpad (e.g., Gear VR's touchpad), by aclick on a remote control (e.g., Oculus Rift remote control), or bypressing a key on a keyboard or using the mouse (e.g., 360 degree videoclient on a PC), or the like.

As shown in FIG. 25, when the application determines that the userdesires to view additional information about a product, it may pause themain video streaming presentation, and may present such additionalinformation, at 2503. FIG. 27 shows an example of additional informationshown by an e-commerce application, for example when a user indicatesinterest on an area of the 360 degree video scene identified by adviewports. Information may be presented, for example, by starting a newvideo streaming session, by presenting a 360 degree photo with productinformation, and/or by showing information and/or clickable urls on anoverlaid head-up display. The url of the new session may be indicatedusing remote periods and xlink, as described herein. Such informationmay be presented until the user indicates that he/she desires tocontinue the main presentation. Such information may be obtained by thee-commerce application from metadata. Metadata may be obtained fromother modules in the system (e.g., from a remote server). Metadata maybe embedded in the segment files received from the server (e.g., withinthe free space (‘free’ or ‘skip’) box of the iso-bmff).

Video analytics may be performed for both 360 degree and rectilinear(e.g., 2D) videos. For example, whether viewers are looking at a productor brand identifier in video advertisements may be determined. Whereattention is drawn within a web page may be determined. In therectilinear (2D) video case, even though the entire video frame mayalways be within the field of view of the user, the eye may be directedtoward different elements on the screen, even on small screens such assmartphones.

FIG. 28 shows an example video streaming system where secondary contentviewports may be used for video analytics. Secondary content viewportsmay be used to specify tracking boxes around products in 360 degreevideo frames and/or in rectilinear (2D) video frames. The streamingclient may determine and report whether the viewer looked at the producttracking boxes. The video analytics may be reported to a server whereinformation from multiple streaming clients may be aggregated forfurther analysis.

Video analytics for 360 degree video may be performed. For 360 degreevideo streaming using DASH, the techniques described herein may beapplied for use with video analytics. For example, a streaming client(e.g., DASH streaming client) may be requested to keep orientationstatistics over a period, part of a period or over the entire streamingsession using the methods described herein. Secondary content viewportsmay be defined in the MPD over a period, part of a period, or over theentire streaming session using the methods described herein. Thestreaming client may be requested to report secondary content viewportstatistics back to a reporting server. Secondary content viewportparameter(s) for the secondary content viewport(s) may be tracked asdescribed herein with respect to FIG. 15. For example, user's headorientation information may be reported to a server. As described here,secondary content viewport statistics may be sent to a server or toanother application in the client device.

Video analytics for rectilinear (e.g., 2D) video may be performed togenerate secondary content viewport parameter(s) such as secondarycontent viewport statistics. For rectilinear video streaming using DASH,the techniques described herein may be applied for use with videoanalytics. For example, a streaming client may be requested to keepsecondary content viewport statistics such as orientation statisticsover a period, part of a period or over the entire streaming session.Secondary content viewports may be defined in the MPD over a period,part of a period, or over the entire streaming session. DASH streamingclient may be requested to report secondary content viewport statisticsback to a reporting server. Secondary content viewport statistics may bekept as described herein with respect to FIG. 15. Eye-trackingtechniques may be used to determine where the user's gaze is on thescreen, for example, using web-cams or mobile device cameras. Ifrectilinear (2D) video is viewed using a HMD, orientation informationfrom the HMD may be used to determine where the user's gaze is on thescreen. The secondary content may be selected based on the secondarycontent viewport statistics that may be generated based on user gazeand/or eye-tracking techniques.

FIG. 29A is a diagram illustrating an example communications system 100in which one or more disclosed embodiments may be implemented. Thecommunications system 100 may be a multiple access system that providescontent, such as voice, data, video, messaging, broadcast, etc., tomultiple wireless users. The communications system 100 may enablemultiple wireless users to access such content through the sharing ofsystem resources, including wireless bandwidth. For example, thecommunications systems 100 may employ one or more channel accessmethods, such as code division multiple access (CDMA), time divisionmultiple access (TDMA), frequency division multiple access (FDMA),orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tailunique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM(UW-OFDM), resource block-filtered OFDM, filter bank multicarrier(FBMC), and the like.

As shown in FIG. 29A, the communications system 100 may include wirelesstransmit/receive units (WTRUs) 102 a, 102 b, 102 c, 102 d, a RAN104/113, a CN 106/115, a public switched telephone network (PSTN) 108,the Internet 110, and other networks 112, though it will be appreciatedthat the disclosed embodiments contemplate any number of WTRUs, basestations, networks, and/or network elements. Each of the WTRUs 102 a.102 b. 102 c, 102 d may be any type of device configured to operateand/or communicate in a wireless environment. By way of example, theWTRUs 102 a, 102 b, 102 c, 102 d, any of which may be referred to as a“station” and/or a “STA”, may be configured to transmit and/or receivewireless signals and may include a user equipment (UE), a mobilestation, a fixed or mobile subscriber unit, a subscription-based unit, apager, a cellular telephone, a personal digital assistant (PDA), asmartphone, a laptop, a netbook, a personal computer, a wireless sensor,a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watchor other wearable, a head-mounted display (HMD), a vehicle, a drone, amedical device and applications (e.g. remote surgery), an industrialdevice and applications (e.g. a robot and/or other wireless devicesoperating in an industrial and/or an automated processing chaincontexts), a consumer electronics device, a device operating oncommercial and/or industrial wireless networks, and the like. Any of theWTRUs 102 a. 102 b, 102 c and 102 d may be interchangeably referred toas a UE.

The communications systems 100 may also include a base station 114 aand/or a base station 114 b. Each of the base stations 114 a, 114 b maybe any type of device configured to wirelessly interface with at leastone of the WTRUs 102 a, 102 b, 102 c, 102 d to facilitate access to oneor more communication networks, such as the CN 106/115, the Internet110, and/or the other networks 112. By way of example, the base stations114 a, 114 b may be a base transceiver station (BTS), a Node-B, an eNodeB, a Home Node B, a Home eNode B, a gNB, a NR NodeB, a site controller,an access point (AP), a wireless router, and the like. While the basestations 114 a, 114 b are each depicted as a single element, it will beappreciated that the base stations 114 a, 114 b may include any numberof interconnected base stations and/or network elements.

The base station 114 a may be part of the RAN 104/113, which may alsoinclude other base stations and/or network elements (not shown), such asa base station controller (BSC), a radio network controller (RNC), relaynodes, etc. The base station 114 a and/or the base station 114 b may beconfigured to transmit and/or receive wireless signals on one or morecarrier frequencies, which may be referred to as a cell (not shown).These frequencies may be in licensed spectrum, unlicensed spectrum, or acombination of licensed and unlicensed spectrum. A cell may providecoverage for a wireless service to a specific geographical area that maybe relatively fixed or that may change over time. The cell may furtherbe divided into cell sectors. For example, the cell associated with thebase station 114 a may be divided into three sectors. Thus, in oneembodiment, the base station 114 a may include three transceivers, i.e.,one for each sector of the cell. In an embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology and mayutilize multiple transceivers for each sector of the cell. For example,beamforming may be used to transmit and/or receive signals in desiredspatial directions.

The base stations 114 a, 114 b may communicate with one or more of theWTRUs 102 a, 102 b, 102 c, 102 d over an air interface 116, which may beany suitable wireless communication link (e.g. radio frequency (RF),microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet(UV), visible light, etc.). The air interface 116 may be establishedusing any suitable radio access technology (RAT).

More specifically, as noted above, the communications system 100 may bea multiple access system and may employ one or more channel accessschemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. Forexample, the base station 114 a in the RAN 104/113 and the WTRUs 102 a.102 b, 102 c may implement a radio technology such as Universal MobileTelecommunications System (UMTS) Terrestrial Radio Access (UTRA), whichmay establish the air interface 115/116/117 using wideband CDMA (WCDMA).WCDMA may include communication protocols such as High-Speed PacketAccess (HSPA) and/or Evolved HSPA (HSPA+). HSPA may include High-SpeedDownlink (DL) Packet Access (HSDPA) and/or High-Speed UL Packet Access(HSUPA).

In an embodiment, the base station 114 a and the WTRUs 102 a, 102 b, 102c may implement a radio technology such as Evolved UMTS TerrestrialRadio Access (E-UTRA), which may establish the air interface 116 usingLong Term Evolution (LTE) and/or LTE-Advanced (LTE-A) and/orLTE-Advanced Pro (LTE-A Pro).

In an embodiment, the base station 114 a and the WTRUs 102 a. 102 b, 102c may implement a radio technology such as NR Radio Access, which mayestablish the air interface 116 using New Radio (NR).

In an embodiment, the base station 114 a and the WTRUs 102 a, 102 b, 102c may implement multiple radio access technologies. For example, thebase station 114 a and the WTRUs 102 a, 102 b, 102 c may implement LTEradio access and NR radio access together, for instance using dualconnectivity (DC) principles. Thus, the air interface utilized by WTRUs102 a. 102 b, 102 c may be characterized by multiple types of radioaccess technologies and/or transmissions sent to/from multiple types ofbase stations (e.g. an eNB and a gNB).

In other embodiments, the base station 114 a and the WTRUs 102 a. 102 b,102 c may implement radio technologies such as IEEE 802.11 (i.e.,Wireless Fidelity (WiFi), IEEE 802.16 (i.e., Worldwide Interoperabilityfor Microwave Access (WiMAX)), CDMA2000, CDMA2000 1×, CDMA2000 EV-DO,Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), InterimStandard 856 (IS-856), Global System for Mobile communications (GSM),Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and thelike.

The base station 114 b in FIG. 29A may be a wireless router, Home NodeB, Home eNode B, or access point, for example, and may utilize anysuitable RAT for facilitating wireless connectivity in a localized area,such as a place of business, a home, a vehicle, a campus, an industrialfacility, an air corridor (e.g. for use by drones), a roadway, and thelike. In one embodiment, the base station 114 b and the WTRUs 102 c, 102d may implement a radio technology such as IEEE 802.11 to establish awireless local area network (WLAN). In an embodiment, the base station114 b and the WTRUs 102 c, 102 d may implement a radio technology suchas IEEE 802.15 to establish a wireless personal area network (WPAN). Inyet another embodiment, the base station 114 b and the WTRUs 102 c, 102d may utilize a cellular-based RAT (e.g. WCDMA, CDMA2000, GSM, LTE,LTE-A. LTE-A Pro, NR etc.) to establish a picocell or femtocell. Asshown in FIG. 29A, the base station 114 b may have a direct connectionto the Internet 110. Thus, the base station 114 b may not be required toaccess the Internet 110 via the CN 106/115.

The RAN 104/113 may be in communication with the CN 106/115, which maybe any type of network configured to provide voice, data, applications,and/or voice over internet protocol (VoIP) services to one or more ofthe WTRUs 102 a, 102 b, 102 c, 102 d. The data may have varying qualityof service (QoS) requirements, such as differing throughputrequirements, latency requirements, error tolerance requirements,reliability requirements, data throughput requirements, mobilityrequirements, and the like. The CN 106/115 may provide call control,billing services, mobile location-based services, pre-paid calling,Internet connectivity, video distribution, etc., and/or performhigh-level security functions, such as user authentication. Although notshown in FIG. 29A, it will be appreciated that the RAN 104/113 and/orthe CN 106/115 may be in direct or indirect communication with otherRANs that employ the same RAT as the RAN 104/113 or a different RAT. Forexample, in addition to being connected to the RAN 104/113, which may beutilizing a NR radio technology, the CN 106/115 may also be incommunication with another RAN (not shown) employing a GSM, UMTS, CDMA2000, WiMAX. E-UTRA, or WiFi radio technology.

The CN 106/115 may also serve as a gateway for the WTRUs 102 a, 102 b,102 c, 102 d to access the PSTN 108, the Internet 110, and/or the othernetworks 112. The PSTN 108 may include circuit-switched telephonenetworks that provide plain old telephone service (POTS). The Internet110 may include a global system of interconnected computer networks anddevices that use common communication protocols, such as thetransmission control protocol (TCP), user datagram protocol (UDP) and/orthe internet protocol (IP) in the TCP/IP internet protocol suite. Thenetworks 112 may include wired and/or wireless communications networksowned and/or operated by other service providers. For example, thenetworks 112 may include another CN connected to one or more RANs, whichmay employ the same RAT as the RAN 104/113 or a different RAT.

Some or all of the WTRUs 102 a, 102 b, 102 c, 102 d in thecommunications system 100 may include multi-mode capabilities (e.g. theWTRUs 102 a, 102 b, 102 c, 102 d may include multiple transceivers forcommunicating with different wireless networks over different wirelesslinks). For example, the WTRU 102 c shown in FIG. 29A may be configuredto communicate with the base station 114 a, which may employ acellular-based radio technology, and with the base station 114 b, whichmay employ an IEEE 802 radio technology.

FIG. 29B is a system diagram illustrating an example WTRU 102. As shownin FIG. 29B, the WTRU 102 may include a processor 118, a transceiver120, a transmit/receive element 122, a speaker/microphone 124, a keypad126, a display/touchpad 128, non-removable memory 130, removable memory132, a power source 134, a global positioning system (GPS) chipset 136,and/or other peripherals 138, among others. It will be appreciated thatthe WTRU 102 may include any sub-combination of the foregoing elementswhile remaining consistent with an embodiment.

The processor 118 may be a general purpose processor, a special purposeprocessor, a conventional processor, a digital signal processor (DSP), aplurality of microprocessors, one or more microprocessors in associationwith a DSP core, a controller, a microcontroller, Application SpecificIntegrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs)circuits, any other type of integrated circuit (IC), a state machine,and the like. The processor 118 may perform signal coding, dataprocessing, power control, input/output processing, and/or any otherfunctionality that enables the WTRU 102 to operate in a wirelessenvironment. The processor 118 may be coupled to the transceiver 120,which may be coupled to the transmit/receive element 122. While FIG. 29Bdepicts the processor 118 and the transceiver 120 as separatecomponents, it will be appreciated that the processor 118 and thetransceiver 120 may be integrated together in an electronic package orchip.

The transmit/receive element 122 may be configured to transmit signalsto, or receive signals from, a base station (e.g. the base station 114a) over the air interface 116. For example, in one embodiment, thetransmit/receive element 122 may be an antenna configured to transmitand/or receive RF signals. In an embodiment, the transmit/receiveelement 122 may be an emitter/detector configured to transmit and/orreceive IR, UV, or visible light signals, for example. In yet anotherembodiment, the transmit/receive element 122 may be configured totransmit and/or receive both RF and light signals. It will beappreciated that the transmit/receive element 122 may be configured totransmit and/or receive any combination of wireless signals.

Although the transmit/receive element 122 is depicted in FIG. 29B as asingle element, the WTRU 102 may include any number of transmit/receiveelements 122. More specifically, the WTRU 102 may employ MIMOtechnology. Thus, in one embodiment, the WTRU 102 may include two ormore transmit/receive elements 122 (e.g. multiple antennas) fortransmitting and receiving wireless signals over the air interface 116.

The transceiver 120 may be configured to modulate the signals that areto be transmitted by the transmit/receive element 122 and to demodulatethe signals that are received by the transmit/receive element 122. Asnoted above, the WTRU 102 may have multi-mode capabilities. Thus, thetransceiver 120 may include multiple transceivers for enabling the WTRU102 to communicate via multiple RATs, such as NR and IEEE 802.11, forexample.

The processor 118 of the WTRU 102 may be coupled to, and may receiveuser input data from, the speaker/microphone 124, the keypad 126, and/orthe display/touchpad 128 (e.g. a liquid crystal display (LCD) displayunit or organic light-emitting diode (OLED) display unit). The processor118 may also output user data to the speaker/microphone 124, the keypad126, and/or the display/touchpad 128. In addition, the processor 118 mayaccess information from, and store data in, any type of suitable memory,such as the non-removable memory 130 and/or the removable memory 132.The non-removable memory 130 may include random-access memory (RAM),read-only memory (ROM), a hard disk, or any other type of memory storagedevice. The removable memory 132 may include a subscriber identitymodule (SIM) card, a memory stick, a secure digital (SD) memory card,and the like. In other embodiments, the processor 118 may accessinformation from, and store data in, memory that is not physicallylocated on the WTRU 102, such as on a server or a home computer (notshown).

The processor 118 may receive power from the power source 134, and maybe configured to distribute and/or control the power to the othercomponents in the WTRU 102. The power source 134 may be any suitabledevice for powering the WTRU 102. For example, the power source 134 mayinclude one or more dry cell batteries (e.g. nickel-cadmium (NiCd),nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion),etc.), solar cells, fuel cells, and the like.

The processor 118 may also be coupled to the GPS chipset 136, which maybe configured to provide location information (e.g. longitude andlatitude) regarding the current location of the WTRU 102. In additionto, or in lieu of, the information from the GPS chipset 136, the WTRU102 may receive location information over the air interface 116 from abase station (e.g. base stations 114 a, 114 b) and/or determine itslocation based on the timing of the signals being received from two ormore nearby base stations. It will be appreciated that the WTRU 102 mayacquire location information by way of any suitablelocation-determination method while remaining consistent with anembodiment.

The processor 118 may further be coupled to other peripherals 138, whichmay include one or more software and/or hardware modules that provideadditional features, functionality and/or wired or wirelessconnectivity. For example, the peripherals 138 may include anaccelerometer, an e-compass, a satellite transceiver, a digital camera(for photographs and/or video), a universal serial bus (USB) port, avibration device, a television transceiver, a hands free headset, aBluetooth® module, a frequency modulated (FM) radio unit, a digitalmusic player, a media player, a video game player module, an Internetbrowser, a Virtual Reality and/or Augmented Reality (VR/AR) device, anactivity tracker, and the like. The peripherals 138 may include one ormore sensors, the sensors may be one or more of a gyroscope, anaccelerometer, a hall effect sensor, a magnetometer, an orientationsensor, a proximity sensor, a temperature sensor, a time sensor, ageolocation sensor; an altimeter, a light sensor, a touch sensor, amagnetometer, a barometer, a gesture sensor, a biometric sensor, and/ora humidity sensor.

The WTRU 102 may include a full duplex radio for which transmission andreception of some or all of the signals (e.g. associated with particularsubframes for both the UL (e.g. for transmission) and downlink (e.g. forreception) may be concurrent and/or simultaneous. The full duplex radiomay include an interference management unit 139 to reduce and orsubstantially eliminate self-interference via either hardware (e.g. achoke) or signal processing via a processor (e.g. a separate processor(not shown) or via processor 118). In an embodiment, the WRTU 102 mayinclude a half-duplex radio for which transmission and reception of someor all of the signals (e.g. associated with particular subframes foreither the UL (e.g. for transmission) or the downlink (e.g. forreception)).

FIG. 29C is a system diagram illustrating the RAN 104 and the CN 106according to an embodiment. As noted above, the RAN 104 may employ anE-UTRA radio technology to communicate with the WTRUs 102 a, 102 b, 102c over the air interface 116. The RAN 104 may also be in communicationwith the CN 106.

The RAN 104 may include eNode-Bs 160 a. 160 b, 160 c, though it will beappreciated that the RAN 104 may include any number of eNode-Bs whileremaining consistent with an embodiment. The eNode-Bs 160 a, 160 b, 160c may each include one or more transceivers for communicating with theWTRUs 102 a, 102 b, 102 c over the air interface 116. In one embodiment,the eNode-Bs 160 a. 160 b, 160 c may implement MIMO technology. Thus,the eNode-B 160 a, for example, may use multiple antennas to transmitwireless signals to, and/or receive wireless signals from, the WTRU 102a.

Each of the eNode-Bs 160 a. 160 b, 160 c may be associated with aparticular cell (not shown) and may be configured to handle radioresource management decisions, handover decisions, scheduling of usersin the UL and/or DL, and the like. As shown in FIG. 29C, the eNode-Bs160 a, 160 b, 160 c may communicate with one another over an X2interface.

The CN 106 shown in FIG. 29C may include a mobility management entity(MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN)gateway (or PGW) 166. While each of the foregoing elements are depictedas part of the CN 106, it will be appreciated that any of these elementsmay be owned and/or operated by an entity other than the CN operator.

The MME 162 may be connected to each of the eNode-Bs 162 a, 162 b, 162 cin the RAN 104 via an Si interface and may serve as a control node. Forexample, the MME 162 may be responsible for authenticating users of theWTRUs 102 a, 102 b, 102 c, bearer activation/deactivation, selecting aparticular serving gateway during an initial attach of the WTRUs 102 a,102 b, 102 c, and the like. The MME 162 may provide a control planefunction for switching between the RAN 104 and other RANs (not shown)that employ other radio technologies, such as GSM and/or WCDMA.

The SGW 164 may be connected to each of the eNode Bs 160 a. 160 b, 160 cin the RAN 104 via the Si interface. The SGW 164 may generally route andforward user data packets to/from the WTRUs 102 a, 102 b, 102 c. The SGW164 may perform other functions, such as anchoring user planes duringinter-eNode B handovers, triggering paging when DL data is available forthe WTRUs 102 a, 102 b, 102 c, managing and storing contexts of theWTRUs 102 a, 102 b, 102 c, and the like.

The SGW 164 may be connected to the PGW 166, which may provide the WTRUs102 a. 102 b, 102 c with access to packet-switched networks, such as theInternet 110, to facilitate communications between the WTRUs 102 a, 102b, 102 c and IP-enabled devices.

The CN 106 may facilitate communications with other networks. Forexample, the CN 106 may provide the WTRUs 102 a. 102 b, 102 c withaccess to circuit-switched networks, such as the PSTN 108, to facilitatecommunications between the WTRUs 102 a, 102 b, 102 c and traditionalland-line communications devices. For example, the CN 106 may include,or may communicate with, an IP gateway (e.g. an IP multimedia subsystem(IMS) server) that serves as an interface between the CN 106 and thePSTN 108. In addition, the CN 106 may provide the WTRUs 102 a, 102 b,102 c with access to the other networks 112, which may include otherwired and/or wireless networks that are owned and/or operated by otherservice providers

Although the WTRU is described in FIGS. 29A-29D as a wireless terminal,it is contemplated that in certain representative embodiments that sucha terminal may use (e.g. temporarily or permanently) wired communicationinterfaces with the communication network.

In representative embodiments, the other network 112 may be a WLAN.

A WLAN in Infrastructure Basic Service Set (BSS) mode may have an AccessPoint (AP) for the BSS and one or more stations (STAs) associated withthe AP. The AP may have an access or an interface to a DistributionSystem (DS) or another type of wired/wireless network that carriestraffic in to and/or out of the BSS. Traffic to STAs that originatesfrom outside the BSS may arrive through the AP and may be delivered tothe STAs. Traffic originating from STAs to destinations outside the BSSmay be sent to the AP to be delivered to respective destinations.Traffic between STAs within the BSS may be sent through the AP, forexample, where the source STA may send traffic to the AP and the AP maydeliver the traffic to the destination STA. The traffic between STAswithin a BSS may be considered and/or referred to as peer-to-peertraffic. The peer-to-peer traffic may be sent between (e.g. directlybetween) the source and destination STAs with a direct link setup (DLS).In certain representative embodiments, the DLS may use an 802.11e DLS oran 802.11z tunneled DLS (TDLS). A WLAN using an Independent BSS (IBSS)mode may not have an AP, and the STAs (e.g. all of the STAs) within orusing the IBSS may communicate directly with each other. The IBSS modeof communication may sometimes be referred to herein as an “ad-hoc” modeof communication.

When using the 802.11ac infrastructure mode of operation or a similarmode of operations, the AP may transmit a beacon on a fixed channel,such as a primary channel. The primary channel may be a fixed width(e.g. 20 MHz wide bandwidth) or a dynamically set width via signaling.The primary channel may be the operating channel of the BSS and may beused by the STAs to establish a connection with the AP. In certainrepresentative embodiments, Carrier Sense Multiple Access with CollisionAvoidance (CSMA/CA) may be implemented, for example in 802.11 systems.For CSMA/CA, the STAs (e.g. every STA), including the AP, may sense theprimary channel. If the primary channel is sensed/detected and/ordetermined to be busy by a particular STA, the particular STA may backoff. One STA (e.g. only one station) may transmit at any given time in agiven BSS.

High Throughput (HT) STAs may use a 40 MHz wide channel forcommunication, for example, via a combination of the primary 20 MHzchannel with an adjacent or nonadjacent 20 MHz channel to form a 40 MHzwide channel.

Very High Throughput (VHT) STAs may support 20 MHz, 40 MHz, 80 MHz,and/or 160 MHz wide channels. The 40 MHz. and/or 80 MHz, channels may beformed by combining contiguous 20 MHz channels. A 160 MHz channel may beformed by combining 8 contiguous 20 MHz channels, or by combining twonon-contiguous 80 MHz channels, which may be referred to as an 80+80configuration. For the 80+80 configuration, the data, after channelencoding, may be passed through a segment parser that may divide thedata into two streams. Inverse Fast Fourier Transform (IFFT) processing,and time domain processing, may be done on each stream separately. Thestreams may be mapped on to the two 80 MHz channels, and the data may betransmitted by a transmitting STA. At the receiver of the receiving STA,the above described operation for the 80+80 configuration may bereversed, and the combined data may be sent to the Medium Access Control(MAC).

Sub 1 GHz modes of operation are supported by 802.11af and 802.11ah. Thechannel operating bandwidths, and carriers, are reduced in 802.11af and802.11ah relative to those used in 802.11n, and 802.11ac. 802.11afsupports 5 MHz, 10 MHz and 20 MHz bandwidths in the TV White Space(TVWS) spectrum, and 802.11ah supports 1 MHz, 2 MHz, 4 MHz. 8 MHz, and16 MHz bandwidths using non-TVWS spectrum. According to a representativeembodiment, 802.11ah may support Meter Type Control/Machine-TypeCommunications, such as MTC devices in a macro coverage area. MTCdevices may have certain capabilities, for example, limited capabilitiesincluding support for (e.g. only support for) certain and/or limitedbandwidths. The MTC devices may include a battery with a battery lifeabove a threshold (e.g. to maintain a very long battery life).

WLAN systems, which may support multiple channels, and channelbandwidths, such as 802.11n. 802.11ac, 802.11af, and 802.11ah, include achannel which may be designated as the primary channel. The primarychannel may have a bandwidth equal to the largest common operatingbandwidth supported by all STAs in the BSS. The bandwidth of the primarychannel may be set and/or limited by a STA, from among all STAs inoperating in a BSS, which supports the smallest bandwidth operatingmode. In the example of 802.11ah, the primary channel may be 1 MHz widefor STAs (e.g. MTC type devices) that support (e.g. only support) a 1MHz mode, even if the AP, and other STAs in the BSS support 2 MHz. 4MHz, 8 MHz, 16 MHz, and/or other channel bandwidth operating modes.Carrier sensing and/or Network Allocation Vector (NAV) settings maydepend on the status of the primary channel. If the primary channel isbusy, for example, due to a STA (which supports only a 1 MHz operatingmode), transmitting to the AP, the entire available frequency bands maybe considered busy even though a majority of the frequency bands remainsidle and may be available.

In the United States, the available frequency bands, which may be usedby 802.11ah, are from 902 MHz to 928 MHz. In Korea, the availablefrequency bands are from 917.5 MHz to 923.5 MHz. In Japan, the availablefrequency bands are from 916.5 MHz to 927.5 MHz. The total bandwidthavailable for 802.11ah is 6 MHz to 26 MHz depending on the country code

FIG. 29D is a system diagram illustrating the RAN 113 and the CN 115according to an embodiment. As noted above, the RAN 113 may employ an NRradio technology to communicate with the WTRUs 102 a, 102 b, 102 c overthe air interface 116. The RAN 113 may also be in communication with theCN 115.

The RAN 113 may include gNBs 180 a. 180 b, 180 c, though it will beappreciated that the RAN 113 may include any number of gNBs whileremaining consistent with an embodiment. The gNBs 180 a, 180 b, 180 cmay each include one or more transceivers for communicating with theWTRUs 102 a, 102 b. 102 c over the air interface 116. In one embodiment,the gNBs 180 a, 180 b. 180 c may implement MIMO technology. For example,gNBs 180 a, 108 b may utilize beamforming to transmit signals to and/orreceive signals from the gNBs 180 a, 180 b, 180 c. Thus, the gNB 180 a,for example, may use multiple antennas to transmit wireless signals to,and/or receive wireless signals from, the WTRU 102 a. In an embodiment,the gNBs 180 a, 180 b. 180 c may implement carrier aggregationtechnology. For example, the gNB 180 a may transmit multiple componentcarriers to the WTRU 102 a (not shown). A subset of these componentcarriers may be on unlicensed spectrum while the remaining componentcarriers may be on licensed spectrum. In an embodiment, the gNBs 180 a.180 b, 180 c may implement Coordinated Multi-Point (CoMP) technology.For example, WTRU 102 a may receive coordinated transmissions from gNB180 a and gNB 180 b (and/or gNB 180 c).

The WTRUs 102 a. 102 b, 102 c may communicate with gNBs 180 a, 180 b,180 c using transmissions associated with a scalable numerology. Forexample, the OFDM symbol spacing and/or OFDM subcarrier spacing may varyfor different transmissions, different cells, and/or different portionsof the wireless transmission spectrum. The WTRUs 102 a, 102 b, 102 c maycommunicate with gNBs 180 a. 180 b, 180 c using subframe or transmissiontime intervals (TTIs) of various or scalable lengths (e.g. containingvarying number of OFDM symbols and/or lasting varying lengths ofabsolute time).

The gNBs 180 a. 180 b, 180 c may be configured to communicate with theWTRUs 102 a, 102 b, 102 c in a standalone configuration and/or anon-standalone configuration. In the standalone configuration, WTRUs 102a. 102 b, 102 c may communicate with gNBs 180 a, 180 b, 180 c withoutalso accessing other RANs (e.g. such as eNode-Bs 160 a, 160 b, 160 c).In the standalone configuration, WTRUs 102 a, 102 b, 102 c may utilizeone or more of gNBs 180 a, 180 b, 180 c as a mobility anchor point. Inthe standalone configuration, WTRUs 102 a. 102 b, 102 c may communicatewith gNBs 180 a, 180 b, 180 c using signals in an unlicensed band. In anon-standalone configuration WTRUs 102 a, 102 b, 102 c may communicatewith/connect to gNBs 180 a, 180 b, 180 c while also communicatingwith/connecting to another RAN such as eNode-Bs 160 a, 160 b, 160 c. Forexample, WTRUs 102 a. 102 b, 102 c may implement DC principles tocommunicate with one or more gNBs 180 a. 180 b, 180 c and one or moreeNode-Bs 160 a, 160 b, 160 c substantially simultaneously. In thenon-standalone configuration, eNode-Bs 160 a, 160 b, 160 c may serve asa mobility anchor for WTRUs 102 a, 102 b, 102 c and gNBs 180 a, 180 b,180 c may provide additional coverage and/or throughput for servicingWTRUs 102 a, 102 b, 102 c.

Each of the gNBs 180 a, 180 b, 180 c may be associated with a particularcell (not shown) and may be configured to handle radio resourcemanagement decisions, handover decisions, scheduling of users in the ULand/or DL, support of network slicing, dual connectivity, interworkingbetween NR and E-UTRA, routing of user plane data towards User PlaneFunction (UPF) 184 a, 184 b, routing of control plane informationtowards Access and Mobility Management Function (AMF) 182 a, 182 b andthe like. As shown in FIG. 29D, the gNBs 180 a, 180 b, 180 c maycommunicate with one another over an Xn interface.

The CN 115 shown in FIG. 29D may include at least one AMF 182 a, 182 b,at least one UPF 184 a,184 b, at least one Session Management Function(SMF) 183 a, 183 b, and possibly a Data Network (DN) 185 a. 185 b. Whileeach of the foregoing elements are depicted as part of the CN 115, itwill be appreciated that any of these elements may be owned and/oroperated by an entity other than the CN operator.

The AMF 182 a, 182 b may be connected to one or more of the gNBs 180 a,180 b, 180 c in the RAN 113 via an N2 interface and may serve as acontrol node. For example, the AMF 182 a. 182 b may be responsible forauthenticating users of the WTRUs 102 a, 102 b, 102 c, support fornetwork slicing (e.g. handling of different PDU sessions with differentrequirements), selecting a particular SMF 183 a, 183 b, management ofthe registration area, termination of NAS signaling, mobilitymanagement, and the like. Network slicing may be used by the AMF 182 a,182 b in order to customize CN support for WTRUs 102 a, 102 b, 102 cbased on the types of services being utilized WTRUs 102 a, 102 b, 102 c.For example, different network slices may be established for differentuse cases such as services relying on ultra-reliable low latency (URLLC)access, services relying on enhanced massive mobile broadband (eMBB)access, services for machine type communication (MTC) access, and/or thelike. The AMF 162 may provide a control plane function for switchingbetween the RAN 113 and other RANs (not shown) that employ other radiotechnologies, such as LTE, LTE-A, LTE-A Pro, and/or non-3GPP accesstechnologies such as WiFi.

The SMF 183 a, 183 b may be connected to an AMF 182 a, 182 b in the CN115 via an Nil interface. The SMF 183 a, 183 b may also be connected toa UPF 184 a, 184 b in the CN 115 via an N4 interface. The SMF 183 a, 183b may select and control the UPF 184 a, 184 b and configure the routingof traffic through the UPF 184 a, 184 b. The SMF 183 a. 183 b mayperform other functions, such as managing and allocating UE IP address,managing PDU sessions, controlling policy enforcement and QoS, providingdownlink data notifications, and the like. A PDU session type may beIP-based, non-IP based, Ethernet-based, and the like.

The UPF 184 a, 184 b may be connected to one or more of the gNBs 180 a.180 b. 180 c in the RAN 113 via an N3 interface, which may provide theWTRUs 102 a, 102 b, 102 c with access to packet-switched networks, suchas the Internet 110, to facilitate communications between the WTRUs 102a, 102 b, 102 c and IP-enabled devices. The UPF 184, 184 b may performother functions, such as routing and forwarding packets, enforcing userplane policies, supporting multi-homed PDU sessions, handling user planeQoS, buffering downlink packets, providing mobility anchoring, and thelike.

The CN 115 may facilitate communications with other networks. Forexample, the CN 115 may include, or may communicate with, an IP gateway(e.g. an IP multimedia subsystem (IMS) server) that serves as aninterface between the CN 115 and the PSTN 108. In addition, the CN 115may provide the WTRUs 102 a, 102 b, 102 c with access to the othernetworks 112, which may include other wired and/or wireless networksthat are owned and/or operated by other service providers. In oneembodiment, the WTRUs 102 a, 102 b, 102 c may be connected to a localData Network (DN) 185 a, 185 b through the UPF 184 a, 184 b via the N3interface to the UPF 184 a, 184 b and an N6 interface between the UPF184 a, 184 b and the DN 185 a. 185 b.

In view of FIGS. 29A-D, and the corresponding description of FIGS.29A-D, one or more, or all, of the functions described herein withregard to one or more of: WTRU 102 a-d, Base Station 114 a-b, eNode-B160 a-c, MME 162, SGW 164, PGW 166, gNB 180 a-c, AMF 182 a-ab, UPF 184a-b, SMF 183 a-b, DN 185 a-b, and/or any other device(s) describedherein, may be performed by one or more emulation devices (not shown).The emulation devices may be one or more devices configured to emulateone or more, or all, of the functions described herein. For example, theemulation devices may be used to test other devices and/or to simulatenetwork and/or WTRU functions.

The emulation devices may be designed to implement one or more tests ofother devices in a lab environment and/or in an operator networkenvironment. For example, the one or more emulation devices may performthe one or more, or all, functions while being fully or partiallyimplemented and/or deployed as part of a wired and/or wirelesscommunication network in order to test other devices within thecommunication network. The one or more emulation devices may perform theone or more, or all, functions while being temporarilyimplemented/deployed as part of a wired and/or wireless communicationnetwork. The emulation device may be directly coupled to another devicefor purposes of testing and/or may performing testing using over-the-airwireless communications.

The one or more emulation devices may perform the one or more, includingall, functions while not being implemented/deployed as part of a wiredand/or wireless communication network. For example, the emulationdevices may be utilized in a testing scenario in a testing laboratoryand/or a non-deployed (e.g. testing) wired and/or wireless communicationnetwork in order to implement testing of one or more components. The oneor more emulation devices may be test equipment. Direct RF couplingand/or wireless communications via RF circuitry (e.g. which may includeone or more antennas) may be used by the emulation devices to transmitand/or receive data.

Although features and elements are described above in particularcombinations, one of ordinary skill in the art will appreciate that eachfeature or element can be used alone or in any combination with theother features and elements. In addition, the methods described hereinmay be implemented in a computer program, software, or firmwareincorporated in a computer-readable medium for execution by a computeror processor. Examples of computer-readable media include electronicsignals (transmitted over wired or wireless connections) andcomputer-readable storage media. Examples of computer-readable storagemedia include, but are not limited to, a read only memory (ROM), arandom access memory (RAM), a register, cache memory, semiconductormemory devices, magnetic media such as internal hard disks and removabledisks, magneto-optical media, and optical media such as CD-ROM disks,and digital versatile disks (DVDs). A processor in association withsoftware may be used to implement a radio frequency transceiver for usein a WTRU, UE, terminal, base station, RNC, or any host computer.

1. A method of presenting secondary content in a 360 degree video,comprising: receiving a secondary content viewport location associatedwith a secondary content viewport within a 360 degree video; receiving arequest for tracking a secondary content viewport viewing parameter forthe secondary content viewport; in response to the request, tracking thesecondary content viewport viewing parameter for the secondary contentviewport based on the secondary content viewport location associatedwith the secondary content viewport and a viewer's viewing orientation;and presenting a secondary content at a secondary content insertionlocation based on the secondary content viewport viewing parameter. 2.The method of claim 1, further comprising: determining a secondarycontent insertion position and a size of the secondary content based onthe secondary content viewport viewing parameter.
 3. The method of claim1, further comprising: determining the secondary content for insertionbased on the secondary content viewport viewing parameter.
 4. The methodof claim 1, further comprising: sending the tracked secondary contentviewport viewing parameter to a secondary content decision server; andin response to the sending, receiving the secondary content insertionlocation and an indication of the secondary content for insertion. 5.The method of claim 4, wherein the tracked secondary content viewportviewing parameter is sent in a secondary content request to thesecondary content decision server.
 6. The method of claim 4, wherein thesecondary content insertion location is selected by the secondarycontent decision server based on the secondary content viewport viewingparameter.
 7. The method of claim 1, wherein the secondary contentviewport is a first secondary content viewport, the secondary contentviewport location is a first secondary content viewport location, andthe method further comprises: receiving a second secondary contentviewport location associated with a second secondary content viewportwithin the 360 degree video; tracking the secondary content viewportviewing parameter for the second secondary content viewport based on thesecond secondary content viewport location and the viewer's viewingorientation; and identifying an interested secondary content viewportbased on the tracked secondary content viewport viewing parameter forthe first and second secondary content viewports, wherein the secondarycontent is presented at the secondary content insertion location basedon the identified interested secondary content viewport.
 8. The methodof claim 1, wherein the secondary content viewport location associatedwith the secondary content viewport and the request for tracking thesecondary content viewport viewing parameter are received in a mediarepresentation description (MPD) associated with the 360 degree video.9. The method of claim 1, wherein the secondary content viewportlocation associated with the secondary content viewport and the requestfor tracking the secondary content viewport viewing parameter arereceived in a media representation description (MPD) associated with the360 degree video, and the method further comprising: receiving anupdated MPD indicating the secondary content for insertion at thesecondary content insertion location.
 10. The method of claim 1, whereinthe secondary content viewport viewing parameter comprises at least oneof view counts, a view duration, or viewing orientation statistics. 11.The method of claim 1, further comprising: determining whether theviewer's viewing orientation falls within the secondary content viewportperiodically; and updating the secondary content viewport viewingparameter for the secondary content viewport based on the determining.12. The method of claim 1, wherein the request for tracking thesecondary content viewport viewing parameter comprises a trackingduration that indicates a duration for tracking the secondary contentviewport viewing parameter, a secondary content viewport viewingidentifier, and a secondary content viewport classification frequency.13. A device for presenting secondary content in a 360 degree video, thedevice comprising: a processor configured to: receive a secondarycontent viewport location associated with a secondary content viewportwithin a 360 degree video; receive a request for tracking a secondarycontent viewport viewing parameter for the secondary content viewport;in response to the request, track the secondary content viewport viewingparameter for the secondary content viewport based on the secondarycontent viewport location associated with the secondary content viewportand a viewer's viewing orientation; and present a secondary content at asecondary content insertion location based on the secondary contentviewport viewing parameter.
 14. The device of claim 13, wherein theprocessor is further configured to: determine a secondary contentinsertion position and a size of the secondary content based on thesecondary content viewport viewing parameter.
 15. The device of claim13, wherein the processor is further configured to: determine thesecondary content for insertion based on the secondary content viewportviewing parameter.
 16. The device of claim 13, wherein the processor isfurther configured to: send the tracked secondary content viewportviewing parameter to a secondary content decision server; and inresponse to the sending, receive the secondary content insertionlocation and an indication of the secondary content for insertion. 17.The device of claim 16, wherein the tracked secondary content viewportviewing parameter is sent in a secondary content request to thesecondary content decision server.
 18. The device of claim 16, whereinthe secondary content insertion location is selected by the secondarycontent decision server based on the secondary content viewport viewingparameter.
 19. The device of claim 13, wherein the secondary contentviewport is a first secondary content viewport, the secondary contentviewport location is a first secondary content viewport location, andthe processor is further configured to: receive a second secondarycontent viewport location associated with a second secondary contentviewport within the 360 degree video; track the secondary contentviewport viewing parameter for the second secondary content viewportbased on the second secondary content viewport location and the viewer'sviewing orientation; and identify an interested secondary contentviewport based on the tracked secondary content viewport viewingparameter for the first and second secondary content viewports, whereinthe secondary content is presented at the secondary content insertionlocation based on the identified interested secondary content viewport.20. (canceled)
 21. The device of claim 13, wherein the secondary contentviewport location associated with the secondary content viewport and therequest for tracking the secondary content viewport viewing parameterare received in a media representation description (MPD) associated withthe 360 degree video, and the processor is further configured to:receive an updated MPD indicating the secondary content for insertion atthe secondary content insertion location. 22-24. (canceled)